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Abstract 

“"n 

'y  Inadequately  managed  technical  risks  have  resulted  in 
setbacks,  failures  and  operational  disasters  in  Department 
of  Defense  programs.  Therefore,  the  purpose  of  this 
was  to  synthesize  a  model  that  epitomizes  a 
strategy  for  the  management  of  technical  risk.  The  model 
was  synthesized  using  the  three-pronged  effort  of:  (1)  a 
literature  search  and  review  to  determine  what  previous  work 
was  done  in  risk  assessment  and  risk  management,  (2)  case 
studies  of  historical,  contemporary  and  prospective  new 
technology  development  programs,  and  (3)  interviews 
predominately  with  Chief  Scientists  at  the  Wright  Research 
and  Development  Center  ;(UROGd  at  Wr ight-Patterson  Air  Force 
Base.  The  model  validation  was  via  reviews  of  the  model 
that  were  made  by  the  stated  interviewees.  If  the  model  is 
used  as  a  technical  management  guide  and  decision  aid  by 
individual  program  or  project  managers  at  all  levels, 
collective  marked  improvement  in  the  technical  risk 
management  throughout  the  Department  of  Defense  may  be 
ach  ieved . 
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A  MODEL  FOR  THE  MANAGEMENT  OF  TECHNICAL  RISK  IN 


NEW  TECHNOLOGY  DEVELOPMENT  PROGRAMS 


I .  Introduction 


Overview 

Presently,  a  large  number  of  military  and  civilian 
development  programs  involve  new  technology.  However,  the 
newer  the  technology  that  is  being  introduced  into  a 
development  project,  the  greater  the  risks  that  may  be 
involved  with  successfully  managing  the  development 
program.  Future  military  weapons  systems  may  be  even  more 
complex.  According  to  General  William  Thurman,  Commandant 
cf  the  Defense  Military  Systems  Management  College,  "These 
[future  weapon]  systems  will  create  management  problems 
that  far  surpass  our  contemporary  management  theory  and 
practice"  (143:12).  This  chapter  introduces  the  concept  of 
technical  risk  and  its  implications  in  the  development  of 
new  technology.  The  specific  problem  for  this  research 
effort,  relevant  assumptions,  and  pertinent  definitions  are 
also  presented. 

Background 

At  any  moment,  national  defense  concerns,  or  suspected 
technological  breakthroughs  by  potential  adversaries,  can 
mandate  that  development  programs  involving  new 
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technologies  be  undertaken  by  the  United  States.  For 
instance,  the  Manhattan  Project  was  undertaken  by  the 
United  States  during  World  War  II  to  build  an  atomic  bomb. 
Albert  Einstein's  1939  letter  to  President  Franklin  D. 
Roosevelt  (Appendix  A)  expressed  concern  over  potential 
German  strides  toward  the  first  atomic  weapon  (41:397). 
Similarly,  after  the  4  October  1957  launch  of  the  Sputnik  I 
satellite  into  Earth  orbit  by  the  Soviet  Union,  the  United 
States  reacted  on  31  January  1958,  by  launching  Explorer  I 
(11:32;  42:91;  145:164).  Since  new  technology  is  by 
definition  largely  unprecedented,  new  technology  programs 
would  incur  much  more  technical  risk  than  even  state-of- 
the-art  technology  programs.  At  present,  a  host  of 
relatively  immature,  but  promising,  new  technologies  and 
technological  opportunities  such  as,  X-ray  laser 
technologies,  and  optical  and  neural  computers  are  evident. 
This  suggests  that  technological  breakthroughs  by  any 
potential  adversary  may  plunge  the  United  States  into  a 
host  of  new  technology  weapon  system  development  programs 
involving  unprecedented  technical  risks  (2:88;  21:76).  For 
example,  recent  Soviet  test  flights  of  an  experimental 
aircraft  that  uses  hydrogen  fuel,  could  eventually  dictate 
that  the  United  States  undertake  an  urgent  development 
project  in  that  area.  Also,  if  the  Soviets  are  able  to 
build  and  deploy  an  X-ray  laser,  then  "the  survivability  of 
American  space  assets  could  be  questioned"  (69:1). 
Similarly,  any  advances  in  neural  computers  by  the  Soviets 
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could  theoret  *cally  provide  them  wi  th  crucial  image  and 
pattern  (i.e.,  target)  recognition  breakthroughs.  Further, 
with  the  recent  development  of  the  optical  equivalent  to 
the  transistor,  an  optical  computer  is  becoming 
increasingly  feasible.  Because  an  optical  computer  would 
use  beams  of  light  instead  of  the  e±ectron  currents  that 
are  used  at  present,  an  optical  computer  may  be  capable  of 
"a  trillion  operations  a  second"  (1:85)  and  of  parallel 
processing.  The  military  implications  of  a  successful 
union  of  neural  end  optical  computer  technologies  would  be 
staggering  (1:85,93;  2:88,94;  105:52,55;  123:40). 

However,  urgent  new  technology  development  projects 
could  arise  from  national  involvement  in,  or  observation  of 
combat  or  terrorist  incidents  anywhere  on  the  globe.  For 
example,  the  development  of  more  advanced  Remotely  Piloted 
Vehicles  (RPV)  for  surveillance  and  penetration,  may 
eventually  be  mandated  by  the  effectiveness  with  which  the 
United  States  and  Israel  employed  RPVs  in  the  Vietnam  War 
and  the  1973  Yom  Kippur  War,  respectively  (53:2-18,22,23; 
115:89) .  Also,  the  rate  at  which  the  "Egyptians  and 
Israelis  shot  down  their  own  aircraft  in  the  1973  Middle 
East  War"  (50:2)  and  the  rate  at  which  the  Israelis  shot 
down  their  own  helicopters  during  the  1982  conflict  with 
Syria,  indicated  that  the  technology  to  identify  friend 
from  foe  (IFF)  is  a  critical  combat  requirement.  Those 
observations  have  prompted  the  United  States  to  replace  the 
ailing  1950's  technology  IFF  systems  with  IFF  systems  that 
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will  be  appropriate  to  the  potential  1990's  combat 
environment  (50:1-4), 

It  is  clear  then  that  technology  breakthroughs  by 
potential  United  States  adversaries  in  a  number  of  emerging 
technologies,  or  international  incidents  that  threaten 
national  security,  could  necessitate  urgent  new-technology 
weapon  development  programs  involving  enormous  technical 
risk.  Many  of  these  technologies,  particularly  "directed 
energy  weapons  .  .  .  may  revolutionize  military  strategy, 

tactics  and  doctrine"  (58:120).  It  is  therefore  essential 
that  an  effective  strategy  for  systematically  reducing  the 
technical  risk  inherent  to  new  technology  development 
projects  be  available.  However  significant  technical  risk 
is  also  incurred  in  performing  major  modifications  to 
systems.  A  United  States  Air  Force  study  concluded  that 
"many  modifications  are  generally  as  difficult  as  the 
original  design"  (111:116). 

Justification 

Department  of  Defense  Directive  5000.1,  regarding 
major  and  non-major  weapons  acquisition  programs, 
acknowledges  that  program  decisions  must  be  made 
commensurate  with  technological  risk  (31: para  3,  para  9a, 
para  9b) ,  yet  it  provides  no  explicit  process  for 
systematically  minimizing  that  technical  risk.  Even 
Military  Standard  499A,  Engineering  Management,  although 
stressing  that  technical  risks  must  be  assessed  and 
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minimized,  provides  no  explicit  strategy  for  achieving 
technical  risk  reduction. 

Although  a  number  of  studies  regarding  technical  risks 
in  military  programs  have  been  done  (10;  132;  144),  many 
examine  technical  risk  mainly  as  a  contributor  to  budget 
overruns.  While  a  recent  defense  document  (36)  provides 
excellent  templates  for  the  management  of  technical  risk  in 
transitioning  from  development  to  production,  this  is  only 
a  fraction  of  the  regime  wherein  technical  risk  is  evident. 
Still  fewer  of  these  documents  examine  technical  risk  from 
a  historical  perspective  for  lessons  learned  in  minimizing 
technical  risk  when  the  technology  in  question  is 
relatively  unprecedented. 

What  appears  to  be  needed  is  an  all-encompassing,  high 
lev,  technical  risk  strategy  that  includes  the  extreme 
case  where  the  development  project  involves  totally  new 
technology.  By  addressing  this  extreme  case,  the  strategy 
should  bound  the  less  extreme  cases  of  development  programs 
that  are  extensions  of  more  established  technologies. 

While  a  number  of  military  documents  (25;  27;  39) 
provide  technical  checklists  for  technical  risk  reduction, 
they  do  not  outline  an  explicit  management  process — a  task 
sequence  and  priority — to  achieve  such  risk  reduction.  As 
a  resul*  ,  the  manager  is  usually  left  on  his  or  her  own 
when  it  comes  to  implementing  a  technical  management 
strategy  and  integrating  existing  technical  checklists  into 
that  strategy.  With  no  explicit  technical  management 
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strategy,  each  technical  manager  must  "re-invent  the  wheel" 
when  it  comes  to  technical  risk  management. 

According  to  Koontz,  strategies  are  so  important  to 
success  that 

.  .  .  the  development  and  communication  strategies  are 
among  the  most  important  activities  of  top  managers  . 

.  .  [the  study  has  shown  that  most  business  failures 
are  due]  to  lack  of  strategy,  or  the  wrong  strategy  . 

.  .  without  an  appropriate  strategy,  appropriately 
implemented,  failure  is  a  matter  of  time.  (98:281) 

Accordingly,  a  top  level  management  strategy  for  technical 
risk  reduction,  particularly  for  projects  involving 
unprecedented  technology,  could  be  of  immense  value  to 
Department  of  Defense  weapons  acquisition. 

Statement  of  the  Problem 

At  present,  it  is  not  well  known  how  to  assess  and 
manage  the  technical  risks  associated  with  weapon  systems 
development  programs  which  involve  state-of-the-art 
technology.  The  technical  risks  are  even  greater  when  the 
development  involves  technology  that  is  unprecedented.  The 
question  for  this  research  was:  How  can  the  technical 
risks  of  development  projects  involving  state-of-the-art  or 
unprecedented  technologies  be  minimized?  Once  this  was 
answered,  the  researcher  developed  a  strategy  for  the 
effective  reduction  of  the  technical  risk  associated  with 
new-technology  development  projects.  A  model  was 
synthesized  to  be  used  as  an  effective  management  strategy 
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and  decision-asking  aid  for  minimizing  technical  risk  in 
development  programs  that  involve  new  technology. 

Investigative  Questions 

The  research  focused  on  the  following  investigative 
questions  to  answer  the  stated  research  question. 

1.  How  is  technical  risk  defined? 

2.  How  is  technical  risk  identified? 

3.  What  are  the  important  factors  for  the  assessment 
of  technical  risk? 

4.  What  are  the  major  pitfalls  to  avoid  in  assessing 
or  managing  technical  risk? 

5.  What  criteria  are  used  to  determine  whether  new 
technology  is  sufficiently  established  for  incorporation 
into  a  development  project? 

6.  How  does  the  Air  Force  Wright  Research  and 
Development  Center  Laboratories  (WRDC)  minimize  the 
technical  risks  associated  with  new  technology? 

Scope 

The  risks  in  a  Department  of  Defense  development 
program  are  usually  divided  into  cost,  schedule  and 
performance  (i.e.,  technical)  risks.  This  thesis  will  be 
concerned  only  with  assessing  and  minimizing  the  technical 
risks  connected  with  the  development  or  incorporation  of 
laboratory-demonstrated  technology.  Consequently,  this 
thesis  will  not  address  the  technical  risks  of  achieving 
basic  research  objectives.  Also,  this  thesis  will  not 
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examine  the  impact  of  unstable  funding  or  wavering  national 
or  political  commitment  to  an  ongoing  technical  development 
effort. 

Limitations 

In  keeping  with  the  stated  scope  and  assumptions  -of 
this  research,  a  number  of  factors  that  have  some  impact  on 
technical  risk  have  not  been  addressed.  Consequently,  the 
model  did  not  specifically  include  these  factors. 
Nevertheless,  such  factors  are  important.  Two  such  factors 
are  national  commitment  and  priority.  Selected  specific 
examples  illustrate  the  importance  of  these  factors.  In 
September  1955  President  Eisenhower  assigned  "the  highest 
national  priority"  (42:79)  to  the  research  and  development 
of  the  Intercontinental  Ballistic  Missile  (ICBM) . 

Similarly,  or.  25  May  1961,  President  John  F.  Kennedy,  in 
announcing  the  goal  of  a  man  on  the  moon,  stated  that  the 
decision  requires  "a  national  commitment  of  scientific  and 
technical  manpower,  materials  and  facilities"  (139:29). 
Clearly,  the  national  priority  and  commitment  to  a 
particular  technical  endeavor  will  often  be  a  determinant 
of  the  resources  that  are  dedicated  to.  This  will  in  turn 
hamper  or  help  the  risk  management  effort.  Similarly,  this 
research,  while  acknowledging  the  eventual  effect  of 
schedule  and  funding  limitations  on  the  technical  risk  of  a 
system  under  development,  did  not  explore  such  effects. 
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The  interviews  that  were  done  as  part  of  the  effort  to 
synthesize  the  technical  risk  management  model  were 
predominately  with  Chief  Scientists  of  WRDC  Laboratoies. 
Although  Mr.  Cannon  of  the  ASD  Office  of  Development 
Planning  was  interviewed,  no  other  cognizant  personnel 
associated  with  a  Systems  Programs  Office  were  interviewed. 
Therefore  the  interviews  reflect  technical  risk  from  a 
largely  laboratory  point  of  view. 

Assumptions 

A  major  assumption  of  this  thesis  is  that  the  cost  and 
schedule  risk  of  a  development  project  or  program  are  ".  . 
primarily  attributable  to  the  occurrence  of  unforeseen 
problems  that  are  usually  technical  in  nature"  (99:223)  .  A 
1982  Defense  Science  Board  Task  Force  determined  that  the 
"causes  of  acquisition  risk  are  technical,  not  managerial" 
(39: Sec  1,3)  since  an  industrial  process,  composed  of 
engineering  and  manufacturing,  is  involved. 

Therefore,  a  fundamental  assumption  of  this  thesis  is 
that  if  the  technical  risk  in  the  program  is  minimized, 
then  the  related  cost  and  schedule  risk  of  the  program  will 
be  likewise  minimized.  Another  assumption  is  that  new 
technology  being  introduced  into  a  development  program  has 
been  proven  feasible  via  laboratory  experiments,  although 
techniques  for  adapting  or  tailoring  the  technology  to  a 
specific  application  may  not  have  been  investigated  in 
depth.  This  assumption  is  important  because  this  thesis 
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primarily  examines  the  technical  risk  of  the  development 
of  new  technology  and  not  the  risks  of  attaining  basic 
research  objectives. 

Further,  it  is  assumed  that  the  user's  requirements 
for  the  system  under  development  have  been  clearly  and 
completely  communicated  to  the  agency  charged  with 
developing  the  equipment  or  weapon  system  in  question. 

Last,  but  not  least,  it  is  assumed  that  new  technology 
approaches  cannot  be  avoided  due  to  certain  specified  high 
performance  requirements,  or  based  on  defense  intelligence 
threat  information. 

Acronyms 

The  acronyms  listed  below  apply  to  this  thesis.  Other 

acronyms  are  identified  in  the  text  as  they  occur. 

ASD  (Air  Force)  Aeronautical  Systems  Division  at 

Wright-Patterson  Air  Force  Base,  Ohio 

DARPA  Defense  Advanced  Research  Projects  Agency 

DOD  Department  of  Defense 

DOE  Department  of  Energy 

GAO  General  Accounting  Office 

NACA  National  Advisory  Committee  for  Aeronautics 

(predecessor  of  NASA) 

MIT  Massachusetts  Institute  of  Technology 

NASA  National  Aeronautics  and  Space  Agency 

NATO  North  Atlantic  Treaty  Organization 

NORAD  North  American  Air  Defense  Command 

SDI  Strategic  Defense  Initiative 
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SPO 


System  Program  Office 

SDIO  Strategic  Defense  Initiative  Organization 

WRDC  Air  Force  Wright  Research  and  Development 

Center  (previously  Air  Force  Wright 
Aeronautical  Laboratories) 

Definitions 

For  the  purpose  of  this  thesis,  the  following 
definitions  apply: 

1.  Modularity — the  design  practice  of  grouping 
hardware  or  software  elements  that  perform  the  same 
particular  function  into  "a  self-contained  unit"  (29:44). 

2.  Monte  Carlo  Method — "Any  procedure  that  involves 
statistical  sampling  techniques  in  obtaining  a 
probabilistic  approximation  to  a  mathematical  or  physical 
problem"  (7:69). 

3.  New  Technology — Any  significant  technical  advance, 
possibly  stemming  from  a  scientific  breakthrough,  that 
deviates  significantly  from  the  established  approaches  in 
the  field  or  which  represents  a  marked  improvement  over  the 
current  technical  approach  or  application. 

4.  Performance  Risk — Synonymous  with  technical  risk. 

5.  Risk — "The  probability  and  consequence  of  not 
achieving  some  defined  program  goal  (such  as  cost, 
schedule,  or  performance)"  (24:11-3). 

6.  Risk  Assessment--"The  process  of  examining  a 
situation  and  identifying  the  areas  of  potential  risk" 
(30:Sec  15) . 
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7.  Risk  Analysis — A  detailed,  systematic  examination 
of  the  source  of  risks,  to  determine  the  probability  and 
consequences  of  adverse  events,  in  order  to  evaluate  or 
develop  alternatives  (30:Sec  XV, 1). 

8.  State-of-the-Art — Technology  that  represents  the 
highest  current  knowledge  in  a  particular  technical 
endeavor.  For  all  practical  purposes,  this  term  is 
synonymous  with  the  term  "new  technology". 

9.  Technical  Risk — The  probability  and  effect  of  not 
attaining  a  technical  (i.e.,  performance)  objective.  The 
term  includes  the  risk  of  fatal  accident  or  injury  to 
personnel  as  a  result  of  system  malfunction  or  failure. 

10.  Technological  Risk — Synonymous  with  technical 

risk. 

11.  Work  Breakdown  Structure — A  project  development 
planning,  control,  and  monitoring  tool  that  breaks  down  a 
system  into  its  component  subsystems,  functions  (e.g., 
software,  hardware)  and  indicates  the  interrelationship 
among  these  components  (118:23). 

Summary 

In  many  cases  high  technical  development  risk  stems 
from  the  newness  of  the  technology  or  from  the  particular 
implementation  despite  the  fact  that  the  feasibility  of 
applying  underlying  scientific  principles  may  have  been 
successfully  demonstrated  by  laboratory  experiments. 
Chapter  I,  introduced  the  concept  of  technical  risk, 
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defined  the  problem,  and  specified  the  objectives  of  the 
research.  Chapter  II  provides  the  methodology  whereby  a 
model  for  technical  risk  reduction  was  synthesized. 

Chapter  III  contains  the  literature  search  and  review 
regarding  risk  in  general,  with  the  emphasis  on  technical 
risk.  Chapter  III  also  contains  pertinent  case  studies  to 
this  research.  Chapter  IV  provides  the  analysis 
considerations  for  synthesis  of  the  model  and  then  presents 
the  model.  Chapter  V  is  the  research  conclusion  with 
recommendations  for  further  research. 
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II .  Methodology 


Introduction 

The  purpose  of  this  thesis  is  to  synthesize  a  model 
for  managing  technical  risks  associated  with  new  technology 
development  programs.  To  develop  such  a  model,  a  three¬ 
pronged  research  effort  was  undertaken  consisting  of:  (1) 
a  literature  search  and  review  to  determine  what  previous 
research  was  done;  (2)  case  studies  of  selected  new 
technology  development  projects  in  order  to  glean  lessons- 
learned;  and  (3)  model  reviews  and  interviews  with  Air 
Force  Wright  Research  Development  Center  (WRDC)  scientists 
to  refine  and  validate  the  model.  Figure  1  summarizes  the 
three-pronged  approach  that  was  taken.  The  inputs  of  the 
scientists  to  the  model  synthesis  were  critical  since  the 
laboratories  are  "the  state-of  the  art  experts  in  many 
areas"  (25:6,  Sec  2)  and  because  the  model  is  primarily 
concerned  with  the  development  of  state-of-the-art 
technology.  Figure  2  indicates  pertinent  specific  case 
studies  that  were  selected. 

Based  on  the  results  of  the  literature  review  and  the 
examination  of  selected  case  histories,  a  model  was 
developed  of  the  sequence  of  steps  that  should  be  followed 
in  order  to  systematically  reduce  the  technical  risks 
involved  with  the  use  of  new  technology  in  development 
projects.  The  primary  model  objective  was  to  provide  a 
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Figure  1.  Synthesis  of  Technical  Risk  Management  Model 
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useful  strategy  and  process  for  reducing  technical  risk. 
That  is,  the  emphasis  in  the  model  was  on  the  macro-level 
of  technical  management  strategy  rather  than  on  the  more 
micro-level  of  technical  principles  and  concepts,  at  the 
electronic  or  mechanical  component  level. 

Literature  Search 

As  a  first  step  in  the  research,  a  literature  search 
was  conducted  via  the  Defense  Technical  Information  Center 
(DTIC)  to  determine  what  previous  work  was  done  in  the  area 
of  risk  assessment  and  management.  However,  the  search  was 
supplemented  by  manual  approaches  as  necessary.  The 
literature  review  initially  covered  the  established 
techniques  for  assessing  risk  in  general.  After  the 
general  risk  management  techniques  were  examined,  the  focus 
shifted  specifically  to  the  management  of  technical  risk. 
The  emphasis  here  was  on  those  methods  that  are  used  to 
minimize  the  risks  involved  with  the  development  of  new 
technology  systems. 

Case  Studies  of  New  Technology  Development  Projects 

This  second  part  of  the  thesis  research  to  synthesize 
a  risk  management  model  examined  significant  major 
historical  new  technology  development  projects.  As  Figure 
2  illustrates,  the  cases  were  generally  grouped  under  one 
of  three  categories:  historical,  contemporary  and 
prospective.  During  the  examination  of  each  of  these 
selected  cases,  the  focus  was  on  the  technical  risk 
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reduction  methods  that  were  required  in  order  to  achieve 
the  development  program  in  question.  The  examination  of 
the  selected  cases  was  important  in  order  to  highlight 
those  unanticipated  pitfalls — and  the  method  of  resolution 
— which  can  be  encountered  in  new  technology  development 
projects.  Ultimately,  the  case  studies  lead  to  the 
formulation  of  technical  management  guidelines  which 
constituted  a  fundamental  strategy  for  the  management  of 
technical  risk.  This  resulted,  in  turn,  in  the  synthesis 
of  a  model  which  epitomized  that  strategy. 

Case  Study  Selection  Criteria.  The  criteria  for 
selecting  the  cases  to  be  examined  were  as  follows; 

1.  The  output  of  the  development  project  must  have 
revolutionized  military  strategy  or  tactics  or  had 
significant  international  (i.e.,  technological) 
ramifications. 

2.  The  technical  principles  (i.e.,  feasibility)  of 
the  development  project  in  question  must  have  been 
demonstrated  prior  to  the  start  of  the  the  development 
effort.  However,  the  actual  development  of  the  technology 
must  have  involved  significant  technical  risk. 

3.  Ongoing  major  defense  development  projects  may  be 
selected  if  sufficient  information  is  available  regarding 
some  technical  risks  inherent  to  those  projects. 

4 .  The  case  need  only  meet  one  of  the  above  criteria 
to  be  selected. 
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For  example,  the  airplane,  the  atomic  bomb  and  the 
Intercontinental  Ballistic  Missile  (ICBM)  are  developments 
that  meet  at  least  one  of  the  stated  criteria.  Therefore, 
these  three  cases  were  candidate  cases  to  be  examined. 

In  the  examination  of  the  selected  cases,  one  emphasis 
was  placed  on  determining  which  actions  were  taken  by  the 
technical  staff  in  question  to  eliminate  or  minimize  the 
anticipated  technical  risks  during  development.  In 
addition,  any  unanticipated  technical  problems  that 
surfaced  during  the  analyses  or  development  were  given 
particular  scrutiny  to  determine  whether  or  not  such 
problems  are  characteristic  of  new  technology  development 
projects,  and  to  determine  how  such  technical  problems  were 
overcome  in  the  specific  case. 

Of  particular  concern  was  whether  one  or  more 
categories  of  unanticipated  technical  risk  were  common  to 
two  or  more  of  the  selected  case  histories.  Because  the 
emphasis  is  on  lessons-learned,  the  case  history 
examinations  that  were  conducted  were  of  sufficient  detail 
to  elicit  the  genesis,  effect  and  implications  of  a 
technical  risk  element  in  question  from  a  management 
perspective.  A  detailed  technical  exposition  of  a  case 
study  was  not  attempted  unless  such  a  discourse  was 
immediately  relevant  to  the  synthesis  of  the  technical  risk 
management  model  or  clearly  illustrated  important  technical 
risk  reduction  factors. 
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Technical  Risk  Management  Model 


Based  on  the  literature  review  and  the  examination  of 
selected  case  histories,  an  appropriate  model  was 
synthesized  for  the  technical  risk  assessment  and 
management  of  new  technology  development  programs. 

Groundrules  for  the  Model.  The  following  groundrules 
governed  the  model  and  guided  the  model  synthesis: 

1.  The  model  shall  be  a  general  technical  risk 
management  model;  it  will  be  adaptable  to  development 
programs  involving  new  technology,  as  well  as  to 
development  programs  involving  older,  more  mature 
technology. 

2.  The  model  will  primarily  emphasize  the  strategy 
and  process  for  minimizing  technical  risk  rather  than  the 
detailed  steps  for  technical  activities  (e.g. ,  circuit 
design) .  However,  reference  to  the  documentation 
containing  relevant  detail  will  be  provided. 

3.  The  model  will  indicate  applicable  military 
documents,  as  appropriate,  to  preclude  unnecessary 
duplication  of  information  contained  in  such  documents. 

4.  The  model  will  indicate  a  preferred  sequence  of 
management  tasks  and  possible  paths  to  take  based  on  the 
result  of  a  previous  task(s)  in  the  sequence. 

5.  The  model  shall  indicate  and  distinguish  between 
mandatory  and  highly  recommended  technical  risk  management 
tasks . 
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6.  The  technical  risk  management  model  will  indicate 
priority  and  precedence  (e.g.,  prerequisites)  of  the 
management  tasks  as  well  as  recommended  simultaneous 
events. 

Some  examples  of  formats  which  were  considered  for  use 
included:  networks,  hierarchical  block  diagrams,  decision 
trees,  and  flowcharts.  However,  the  model  format  was  not 
limited  to  these  since  formats  were  selected,  combined  and 
supplemented  to  provide  appropriate  model  detail,  clarity 
and  comprehensiveness  (per  these  stated  groundrules) .  The 
case  histories  were  supplemented  in  the  text  with  specific 
examples  of  technical  risk  situations,  and  management 
techniques  elicited  from  topics  other  than  the  selected 
case  histories. 

Interviews  of  Wright  Research  and  Development  Center  (WRDC) 
Scientists 

The  third  part  of  the  effort  to  synthesize  a  technical 
risk  management  model,  consisted  of  structured  interviews 
with  the  chief  scientist  at  each  of  the  five  respective 
laboratory  divisions  of  the  Air  Force  Wright  Research 
Development  Center  (WRDC)  at  Wright  Patterson  Air  Force 
Base.  The  interviews  were  face— to— face  and  were  conducted 
only  after  a  preliminary  model  had  been  synthesized  based 
on  the  literature  review  and  case  studies. 

The  primary  WRDC  candidates  for  the  interview  was  the 
Chief  Scientist  in  each  of  the  major  divisions  of  AFWAL. 

If  the  Chief  Scientist  was  unavailable  for  the  interviews, 
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then  the  highest  ranking  individual,  civilian  or  military 
having  at  least  a  master's  degree  in  a  technical  discipline 
(e.g.,  engineering,  science,  mathematics)  was  to  be 
selected  for  the  interview. 

Interview  Objectives.  The  primary  objective  of  the 
interviews  was  to  have  the  interviewee  review  and  comment 
on  the  preliminary  technical  risk  management  model.  All 
interviewees  were  given  a  minimum  of  six  working  days  to 
review  the  model.  A  secondary  objective  of  the  interviews 
was  to  determine  what  techniques  these  scientists  consider 
crucial  for  technical  risk  reduction  in  new-technology 
development  projects,  and  to  determine  what  techniques  WRDC 
uses  for  managing  technical  risk.  Comments  and  suggestions 
of  the  experts  were  incorporated  into  a  revised  model  as 
necessary. 

Interview  Questions.  For  the  structured  part  of  the 
interview,  a  list  of  questions  were  developed  which  were 
asked  of  each  chief  scientist.  The  questions  asked 
attempted  to  elicit  the  following  information: 

1.  Technical  and  managerial  experience  of  the  person 
being  interviewed. 

2.  Examples  of  technical  risk  arising  from  new 
technology . 

3.  The  approach  that  is  used  by  their  laboratory 
and/or  organization  to  identify  technical  risk. 

4.  The  approach  that  is  used  by  their  laboratory 
and/or  organization  to  manage  technical  risk.  This 
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included  a  determination  of  the  approach  used,  if  any  to 
monitor  the  level  of  risk  reduction. 

5.  Any  specific  quantitative  and  qualitative 
approaches  that  are  being  used  for  technical  rirk 
management. 

6.  Key  documents,  references  or  regulations  that 
he/she  cites  as  key  guidance  for  providing  risk  assessment 
and  risk  management  procedures. 

The  unstructured  part  of  the  interview  consisted  of 
general  question  to  allow  the  interviewee  to  comment  on 
relevant  technical  risk  management  items  that  may  not  have 
been  touched  on  during  the  structured  interview.  The 
specific  questions  that  were  asked  on  the  interview  are  in 
Appendix  B. 

WKDC  Review  of  Preliminary  Model. 

After  the  interview,  the  interviewees  were  asked  to 
provide  their  comments  on  the  preliminary  technical  risk 
management  model.  In  particular,  the  interviewees  were 
asked  to  comment  on  the  adequacy,  utility,  limitations  of 
the  model  and  to  recommend  specific  ways  that  the  mode., 
could  be  improved.  The  interviewees  were  encouraged  to 
indicate  the  recommended  revisions  by  marking  the  model. 
The  model  revision  was  then  reviewed  a  second  time  by  the 
experts  to  validate  the  final  product. 
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WRDC  Review  of  Revised  Model 


The  model  was  revised  based  on  the  scientists'  review, 
comments  and  recommendations.  To  verify  that  the 
interviewees  concurred  with  the  revised  model,  a  follow-up 
visit  was  conducted  to  provide  the  interviewees  with  one 
final  chance  to  comment.  In  the  event  that  che  individual 
who  provided  a  comment  was  unavailable  for  the  follow-up 
visit  a  copy  of  the  revised  model  was  left  for  review. 

In  the  event  that  any  conflicting  comments  or 
recommendations  pertaining  to  the  model  could  not  be 
resolved,  the  reason  for  the  conflict  was  so  noted  and 
indicated  on  the  model  (e.g.,  by  dashed  lines). 


Model  Review  and  Interviews 

An  attempt  was  made  to  interview  each  of  the  Chief 
Scientists  of  the  Wright  Research  Development  Center 
Laboratories.  Each  interviewee  also  reviewed  the 
preliminary  model.  The  interviewees/model  reviewers  with 
their  respective  position  titles  indicated,  are: 


Dr.  Keith  Richey  Technical  Director  cf  WRDC 

x. 59400 


Dr.  Harris  Burte  Chief  Scientist,  WRDC 

x. 52738  Materials  Laboratory 


Dr.  Jim  Olsen  Chief  Scentist,  WRDC 

x. 57239  Flight  Dynamics  Laboratory 


Dr.  Arnold  Mayer 
x. 53311 


Chief  Assistant  for 
Research  and  Technology, 
Vehicle  Subsystems  Division 
of  WRDC  Flight  Dynamics 
Laboratory 
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Dr.  Squire  Brown 
x. 52377 

Mr.  Jack  Cannon 
X.  55877 


Chief  Scientist,  WRDC 
Technology  Division 

Technical  Director  of 
Development  Planning, 
ASD  Office  of  Plans 


Dr.  Jess  Riles 
x. 53627 


Chief  Scientist,  WRDC 
Avionics  Laboratory 


Mr.  C.J.  Cosenza  Chief,  Advanced  Development 

x, 55393  Division,  WRDC  Technology 

Exploitation  Directorate. 


The  WRDC  Laboratory  Chief  Scientists  shown,  reviewed 
the  model  and  provided  an  interview.  For  ease  of  reviewing 
(or  transcribing)  the  interviews,  a  tape  recorder  was  used 
during  the  interviews  with  the  permission  of  each 
interviewee.  Dr.  Curran,  Chief  Scientist  of  the  WRDC 
Aerospace  Propulsion  and  Power  Laboratory,  was  unavailable 
during  the  period  of  the  interviews.  Dr.  Olsen  and  Dr. 
Mayer  were  interviewed  jointly.  Mr.  Cannon  and  Mr. 

Cosenza,  were  interviewed  and  they  also  reviewed  the  model; 
they  were  both  recommended  by  one  or  more  chief  scientist. 
The  phone  extensions  shown  are  those  on  Wright-Patterson 
Air  Force  Base,  Ohio. 
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Ill .  Literature  Review 


Background 

In  1981,  US  Air  Force  Brigadier  General  William 
Thurman,  Commandant  of  the  Defense  Systems  Military 
College,  stated: 

There  is  an  impressive  and  growing  list  of  failures  in 
large-scale  advanced  technology  programs.  Many  of 
these  failures  involve  military  programs,  but  there 
have  also  been  numerous  failures  in  non-military 
projects.  We  know  less  than  we  think  we  do  about  the 
management  process  by  which  new  technology  is 
converted  into  operating  systems.  (143:12) 

Even  though  a  new  technology  may  have  been 
demonstrated  in  a  laboratory,  the  risks  of  failure  in 
developing  (i.e.,  applying)  that  technology  may  be  high. 
This  is  because  information  on  the  use  and  the  limitations 
of  the  new  technology  in  a  particular  application,  may  be 
inadequate  or  unavailable.  Such  limited  information 
imposes  uncertainty,  and  the  associated  risk  of  failing  the 
development  project  in  question.  Specific  steps  are 
required  to  systematically  minimize  the  attendant  technical 
development  risks  associated  with  new  technology  (80:148; 
143:12) . 

The  United  States'  Manhattan  Project  is  perhaps  the 
premier  example  of  a  new  technology  development  program. 

The  Manhattan  Project  was  undertaken  during  World  War  II  to 
design  and  build  an  atomic  bomb  based  on  the  principle  of 
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atomic  fission  that  was  successfully  achieved  in  the 
laboratory.  The  development  of  an  atom  bomb  involved 
considerable  technical  risk  (80:180-181).  A  contemporary 
example  of  a  major  technical  undertaking  that  relies 
heavily  on  the  state-of-the-art  technology,  and  that 
involves  concomitant  high  technological  risk  is  the  United 
States'  prospective  Strategic  Defense  Initiative  (SDI) — 
colloquially  referred  to  as  the  "Star  Wars"  program.  In 
his  SDI  announcement  on  23  March  1983,  President  Ronald 
Reagan  challenged  the  defense  technological  community  to 
develop  the  technology  which  could  "...  intercept  and 
destroy  strategic  ballistic  missiles  before  they  reached 
our  own  soil  or  that  of  our  allies"  (97:8).  According  to 
Dr.  Donald  Ulrich,  senior  program  director  for  the  Air 
Force  Office  of  Scientific  Research,  a  significant  amount 
of  the  technical  risk  reduction  in  the  SDI  program  depends 
on  the  availability  of  advanced  materials  suitable  for  the 
formidable  space  environment  (84:10;  97:8,9). 

Another  project  which  would  involve  considerable 
technical  risk  is  the  transatmospheric  flight  vehicle, 
announced  in  President  Reagan's  1986  State  of  the  Union 
Address.  That  aircraft,  called  the  National  Aero-Space 
Plane  (NASP)  may  be  able  to  travel  from  California  to  the 
Orient  within  two  hours,  have  the  capability  to  travel  at 
"speeds  (above  Mach  6)  in  the  atmosphere"  (133:13),  enter 
space,  and  then  re-enter  the  atmosphere  for  a  runway 
landing  (140:67) . 
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State-of-the-art  technologies  such  as  Very-High-opeed- 
Integrated-Circuits  (VHSIC) ,  being  pursued  by  the  Air  Force 
and  Navy,  present  significant  technical  risk — especially 
where  semiconductor  material  processing  and  fabrication  are 
concerned.  Other  high  technology  development  efforts  in 
the  areas  of  high-power  lasers  and  microwave  weapon 
technology  also  involve  high  technical  risk.  Despite  the 
high  technical  risks,  these  technologies— and  others — are 
being  pursued  because  they  are  crucial  to  a  strong  national 
defense  (16:50-53;  33:1,2;  34:1,2;  35:1,2). 

To  underscore  the  importance  of  adequately  identifying 
and  managing  technical  risk,  and  to  illustrate  that 
technical  risk  may  also  be  inherent  to  established 
technologies,  one  can  cite  two  recent  false  alarms  at  the 
North  American  Air  Defense  Command  (NoRAD) . 

On  June  3,  1980,  false  attack  indications  were  caused 
by  a  faulty  component  in  a  communications  processor 
computer.  On  June  6,  1980,  false  attack  indications 
were  once  again  caused  by  the  faulty  component  during 
operational  testing.  (61:3) 

Considering  the  crucial  defense  role  of  NORAD,  it  is 
clear  that  any  technical  performance  deficiencies  inherent 
to  NORAD  attack  warning  equipment,  including  false  alarms, 
jeopardize  national  defense  (61:3,13,22). 

Definition  of  Risk 

Webster  defines  risk  as  "the  possibility  of  loss, 
injury,  disadvantage  or  destruction"  (146:1961).  Still, 
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other  definitions  of  risk  focus  on  safety  considerations. 
Risk  has  been  defined  as  "The  probability  of  loss  or  injury 
to  people  and  property"  (89:9).  In  fact,  risk  is  often 
associated  with  a  mathematical  probability  or  a  likelihood 
of  occurrence. 

However,  it  is  acknowledged  that  "there  is 
considerable  overlap  (and  often  confusion)  between  the 
terms  Reliability,  Safety,  Hazard  and  Risk"  (89:9).  For 
instance,  the  risk  of  fatal  accident  in  an  aircraft  was 
quantified  by  the  early  1960 's  as  being  one  fatal  accident 
out  of  every  million  flights  (89:1).  This  is  consistent 
with  the  definition  that  risk  is  "an  expression  of  the 
possibility  of  a  mishap  in  terms  of  hazard  severity  and 
hazard  probability"  (39:3).  In  the  Department  of  Defense 
(DOD)  risk  has  been  defined  as  "a  potential  occurrence  that 
would  be  detrimental  to  plans  or  programs"  (30:1,  Sec  15). 

Haimes  distinguishes  between  risky  and  uncertain 
situations  as  being,  respectively,  those  in  which  "the 
potential  outcomes  can  be  described  by  reasonably  well 
Known  probability  distributions"  (85:217),  and  those  in 
which  they  cannot.  Nevertheless  Haimes  acknowledges  that 
the  term  "risk"  can  be  used  to  "connote  situations  of  both 
risk  and  uncertainty"  (85:217).  However,  Golden  and  Martin 
assume  that  "uncertainty  and  risk  can  be  treated  as 
synonymous  terms"  (80:148).  However,  they  have  divided 
risk  into  the  four  categories  of  "environmental, 
functional,  informational  and  technical  risk"  (80:148). 
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These  four  categories  they  have  subdivided  even  further  to 
create  a  hierarchy  of  risk  categorization.  A  Rand  study 
proposes  a  similarly  structured  "hierarchy  of  uncertainty" 
(144:24)  with  a  probability  distribution  associated  with 
the  success  of  each  component  of  each  level  of  the  system 
hierarchy  in  question.  According  to  the  Rand  study,  the 
collective  probability  of  system  success  can  then  be 
obtained  from  these  individual  system  components  by  using 
the  techniques  of  Monte  Carlo  Simulation  and  Propagation  of 
Error  ( 144 : 24 ) . 

Similarly,  in  the  Department  of  Defense,  weapon  system 
development  risk  has  typically  been  divided  into 
performance,  cost  and  schedule  risks,  although  these  risks 
are  actually  interrelated  (28:27;  151:97).  A  Defense 
Systems  Management  college  handbook  on  risk  assessment 
techniques  defines  risk  as  the  "probability  and  consequence 
of  not  achieving  some  defined  program  goal  (such  as  cost, 
schedule,  or  technical  performance)"  (24:3,  Sec  II).  In 
summary,  most  definitions  of  risk  are  centered  around  the 
concept  of  uncertainty  since  in  the  presence  of  certainty, 
there  is  no  risk  (80:148;  143:38). 

Technical  Risks 

Technical  risks  are  "those  problems  or  uncertainties 
that  may  hinder  the  achievement  of  design  and  development 
goals"  (58:1)  of  a  weapon  system.  The  technical  risk  in 
complex  weapon  system  development  stems  from  unknowns 
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connected  with  " .  .  .  inadequate  knowledge  of  the  basic 
technology  or  its  specific  implementation”  (143:12). 

The  unknowns  in  a  project  may  be  subdivided  into 
recognized  unknowns  and  unrecognized  unknowns.  A 
recognized  unknown  is  an  item  for  which  an  individual  is 
aware  that  the  present  information  or  knowledge  is  absent 
or  inadequate.  However,  an  unrecognized  unknown  is  an  item 
of  which  an  individual  is  not  even  aware  is,  or  will  be,  a 
pertinent  factor  for  technical  risk  reduction  for  the 
project  or  endeavor  in  question.  Therefore  the  planning  to 
effectively  handle  unknowns — inf ormation  risk — is  important 
for  systematic  risk  reduction  (132:38).  Indeed,  the 
performance  risk  associated  with  new  technology  stems  from 
uncertainty.  Rowe  cautions  that  ir.  addition  to  unknowns 
connected  with  pushing  the  state-of-the-art,  technological 
risk  is  also  due  to  " .  .  .a  major  gap  between  an 
organizations'  area  of  expertise  and  what  is  required  to 
perform  effectively"  (132:40).  In  fact,  Shannon  states 
that  actual  project  failures  have  usually  been  traced  to 
either 

1.  Failure  to  seek  the  help  of  appropriate 
specialists. 

2.  Failure  to  ask  the  specialists  the  right  questions 
at  the  right  times. 

3.  Failure  to  heed  the  advice  of  the  specialist  after 
it  is  given.  (137:83) 
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Any  plans  therefore  should  specifically  address  the 
availability,  access  to,  and  utilization  of  specified 
technical  experts  to  minimize  technical  risk  (79:321). 

Risk  Assessment 

Risk  assessment  is  "the  process  of  examining  a 
situation  and  identifying  the  areas  of  potential  risk" 

(30: Sec  15,1).  The  risk  assessment  is  usually  followed  by 
a  risk  analysis  wherein  the  probabilities  and  consequences 
of  particular  events  are  examined  and  quantified.  In  the 
1940 's  technical  risk  management  focused  on  the  improving 
reliability  through  the  enhancement  of  quality  with  better 
design  and  materials  (89:2).  In  1954,  Air  Force  Brigadier 
General  Bernard  A.  Schriever  was  given  authority  to  develop 
"the  free  world's  first  ICBM  (Intercontinental  Ballistic 
Missile],  an  immensely  complex  technical  task"  (9:201). 
However,  ICBM  accidents  drove  the  formalization  of  safety 
studies  as  part  of  system  design: 

The  emergence  of  a  Systems  Safety  Study  as  an 
independent  separate  activity  was  first  mandated  by 
the  Air  Force  in  1962  following  disastrous  accidents 
at  four  ICBM  missile/silo  complexes.  (89:3) 

Ultimately,  this  led  in  1969  to  MIL-STD-882 ,  System 
Safety  Programs  for  Systems  and  Equipment:  Requirements 
for,  as  a  documented  safety  approach  (81:75;  89:4).  In  the 
1960 's,  due  to  the  development  of  the  Intercontinental 
Ballistic  Missiles  and  manned  rocket  programs  such  as 
Project  Mercury  and  Project  Gemini,  there  was  enormous 
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emphasis  on  technical  performance  and  on  the  reduction  of 
technical  risk.  This  was  especially  the  case  because  such 
rocket  launches  could  not  be  redone  once  the  rocket  left 
the  launch  pad.  It  was  in  1961  that  H.  A.  Watson  of  Bell 
Telephone  Laboratories  originated  the  Missile  launch 
control  system.  Fault  Tree  Analysis  remains  an  important 
and  widely  used  technique  for  technical  risk  assessment 
(89:3,45).  General  William  Thurman  stated  that 


Formal  recognition  of  risk  and  uncertainty  analysis  in 
DOD  resulted  from  a  31  July  1969  memorandum  from  David 
Packard  to  the  DEP  SEC  DEF  [Deputy  Secretary  of 
Defense)  identifying  problem  areas  in  the  weapon 
system  acquisition  process.  (143:12) 


The  WASH  1400,  A  Reactor  Safety  Study,  which  was 
commissioned  by  the  United  States  Atomic  Energy  Commission 
and  completed  in  1974,  was  a  comprehensive  risk  assessment 
of  nuclear  facilities.  That  study,  led  by  Professor  N. 
Rasmussen,  remains  a  landmark  in  risk  assessment  and  risk 
analysis  methodology.  Many  of  the  qualitative  and 
quantitative  risk  assessment  techniques  used  in  the  WASH 
1400  study  are  widely  used  today  (89:4,19-24).  The  United 
States  Air  Force  Systems  Command  specifies  that  the  general 
steps  for  a  technical  risk  analysis  are  as  follows: 


1.  Form  a  Risk  Analysis  Task  Force 

2.  Identify  the  objective 

3.  Specify  critical  events 

4.  Develop  contingencies  for  each  event 

5.  Construct  program  network 

6.  Collect  data 

7.  Evaluate  network 

8.  State  conclusions.  (25:Sec  8,7) 
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Henley  and  Kumamoto  outlined  a  three  phase  methodology 
for  conducting  a  technical  risk  analysis  as  outlined  in 
Figure  3.  Similarly,  Haimes  has  defined  risk  assessment  as 
a  process  that  includes  the  five  steps  of  "risk 
identification,  risk  quantification,  risk  evaluation,  risk 
acceptance  and  risk  aversion,  and  risk  management” 

(85:217) . 

Qualitative  Techniques  of  Risk  Assessment 

Qualitative  risk  assessment  techniques  are  those 
techniques  which  rely  heavily  on  opinions  and  judgements  of 
experts  rather  than  on  explicit  empirical  data.  Therefore, 
the  experts  must  have  detailed  technical  knowledge  of  the 
purpose  and  function  of  the  system  in  question  and  of  the 
environment  in  which  the  system  must  operate  (141:77). 

Some  clear  indications  of  technical  expertise  are  the 
holding  of  positions  in  national  scientific  organizations, 
on  editorial  boards  for  key  technical  journals,  and  the 
holding  of  research  contracts  awarded  by  the  government 
(88:3-5).  The  detailed  knowledge  of  experts  is  especially 
important  because  it  is  often  necessary  for  trade-off 
decisions  to  be  made  "among  conflicting  .  .  .  objectives 
and  attributes"  (85:217).  In  particular,  appropriate  and 
early  laboratory  involvement  is  crucial  for  accurate 
assessment  of  technical  risk,  since  the  laboratories  are 
"the  state-of-the-art  experts  in  many  areas"  (25:6,  Sec  2)  . 
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PHASE  I:  DEFINE  THE  SYSTEM  | 

I 

Step  1.  IDENTIFY  THE  HA2ARQ  (Is  it  toxic  ala?.'  j 

explosion,  a  fire...?) 

Step  2.  IDENTIFY  THE  PARTS  OF  THE  SYSTEM  WHICH  GIVE 
RISE  TO  THE  HA2ARDS.  (Does  it  involy.  the 
chemical  reactor,  the  storage  tank,  the  power 
plant?) 

Step  3.  BOUND  THE  STUDY  (Will  it  include  detailed  studies  of 
risks  from  sabotage,  adversary  action,  war,  public 
utility  failures,  lightning,  earthquakes...?) 

PHASE  II:  IDENTIFY  THE  ACCIDENT  SEQUENCES,  EVENT  TREES,  FAULT 
TREES 

PHASE  III:  CONSEQUENCE  ANALYSIS 


Figure  3.  Risk  Analysis  Methodology  (89:19-21,24,29) 
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Harmon  states  that  the  three  characteristics  of 


diversity,  depth,  and  breadth  of  knowledge  are  important 
for  "a  good  panel  of  experts"  (88:4).  To  be  diverse  the 
panel  should  be  comprised  of  multidisciplinary  experts 
(e.g. ,  logistics,  engineering,  safety,  reliability,  etc.) 
in  order  that  multiple  aspects  of  the  problem  or  system  in 
question  can  be  completely  examined.  For  depth,  at  least 
one  expert  should  be  present  with  profound  knowledge  in 
each  major  scientific  or  engineering  field  (29:58,59;  88:3- 
5) .  Finally,  for  breadth,  there  should  be  some  "systems 
experts  .  .  .  [since  they  ore]  accustomed  to  thinking  .  .  . 

in  terms  of  the  interactions  of  various  subsystems"  (88:4). 

Design  Reviews.  Often,  experts  are  assembled  in  a 
conference  or  committee  to  collectively  assess  the 
technical  risks  of  the  system  or  event  in  question  and  to 
recommend  actions  to  prevent  or  reduce  technical  risk  by 
appropriate  design.  These  conferences  usually  constitute 
one  or  more  formal  design  reviews  of  a  system.  The  conduct 
and  content  of  design  reviews  for  weapon  systems  has  been 
formalized  in  the  US  Air  Force  in  Technical  Reviews  and 
Audits  for  Systems,  Equipments,  and  Computer  Software.  MIL- 
STD-1521B  (28:1-123;  38:A-7).  The  assessment  of  risk 
should  also  include  dialogue  with  the  intended  users  and/or 
customers  of  the  system  in  question,  to  include  their 
attendance  at  design  reviews,  since  such  information  could 
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enhance  the  technical  risk  assessment  and  reduction  effort 
(30:2, Sec  XV;  89:19,24). 

However,  the  knowledge  of  even  experts  is  incomplete 
therefore  "technical  errors  and  inconsistencies'*  (132:40) 
are  still  possible.  Fox  stresses  that  technical  personnel 
can  be  overly  optimistic  regarding  what  is  within  the  state 
of  the  art  and  that  this  over-optimism  may  indirectly  lead 
to  technical  risk  since  immature  technological  approaches 
may  be  pursued  (48:128).  For  example,  a  GAO  evaluation 
cited  that  problems  in  the  development  of  the  Inertial 
Upper  Stage  used  to  boost  satellites  from  low  earth  orbit, 
were  actually  due  to  "the  main  contractor's  underestimating 
the  technical  complexity  of  the  Inertial  Upper  Stage" 
(49:61).  Even  experts  very  close  to  a  prospective 
technical  project  may  disagree  on  the  technical  risks.  For 
instance,  Mr.  Roy  Woodruff  allegedly  resigned  from  the 
Lawrence  Livermore  National  Laboratory  late  in  1985  because 
he  claimed  that  "overly  optimistic"  (69:6)  claims  were 
being  made  by  Dr.  Edward  Teller  and  Dr.  Lowell  Wood 
regarding  the  development  potential  for  the  X-ray  laser 
(69:1-6,10;  100:52).  The  stated  Dr.  Teller  is  the  eminent 
scientist  who  has  been  called  the  "father  of  the  Hydrogen 
Bomb"  (87:7;  100:52). 

Delphi  Technique.  For  adequate  risk  assessment,  steps 
must  also  be  taken  to  minimize  certain  negative  group 
processes  such  as  external  or  internal  pressure  for  a 
consensus  among  the  experts,  in  order  to  ensure 


objectivity.  Accordingly,  the  opinions  of  one  or  more 
dissenting  experts  in  such  a  conference  must  be  given 
careful  consideration.  The  Delphi  technique,  whereby 
expert  inputs  are  obtained  in  writing  without  a  conference 
among  the  experts,  may  be  employed  to  solicit  risk  factors 
or  assessments  that  are  almost  devoid  of  group  pressure  to 
conform  to  a  particular  viewpoints.  Instead  the  Delphi 
technique  arrives  at  a  consensus  among  the  experts  by  "an 
anonymous  process"  (136:643)  whereby  several  cycles  of 
written  questions  and  written  sunmary  of  anonymous 
responses  (and  the  accompanying  rationale)  are  presented  to 
each  of  the  experts.  Based  on  the  anonymous  responses  each 
expert  is  allowed  to  modify  his/her  answer  in  successive 
cycles.  Ultimately,  a  consensus  of  the  technical  risks  are 
achieved  (136:643;  141:77). 

To  conduct  a  qualitative  risk  assessment,  the  experts 
must  define  the  system  boundaries  and  broadly  identify 
potential  system  malfunctions  or  catastrophic  failure 
(e.g.,  explosion,  fire,  toxicity).  Then,  the  respective 
system  components  whose  failure  can  ultimately  result  in  a 
particular  hazard  or  malfunction,  are  identified.  Finally, 
the  scope  of  the  study  must  be  appropriately  limited. 
Checklists  providing  some  possible  system  hazards  and 
malfunctions  for  consideration,  are  often  used  to 
facilitate  the  experts'  qualitative  risk  assessment.  A 
checklist  used  by  Boeing  Aircraft  Company  is  presented  in 
Figure  4.  If  the  stated  qualitative  risk  assessment 
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Hazardous  Energy  Sources 

t.  Fuels 

2.  Propellant* 

3.  Initiators 

4.  Explosive  charges 

5.  Charged  electrical 
capacitors 

6.  Storage  batteries 

7.  Static  electrical  charges 

8.  Pressure  container* 

9.  Spring-loaded  devices 

10.  Suspension  Systems 


Hazardous  Processes  and 
Events 

1.  Acceleration 

2.  Contamination 

3.  Corrosion 

4.  Chemical  dissociation 

5.  Electrical 
shock 
thermal 

inadvertent  activation 
power  source  failure 
electromagnetic  radiation 

6.  Explosion 

7.  fire 

8.  Heat  and  temperature 
high  temperature 
low  temperature 
temperature  variations 

9.  Leakage 


Figure  4.  Checklists  of 


1 1.  Gas  generators 

12.  Electrical  generators 

13.  rf  energy  sources 

14.  Radioactive  energy 
sources 

15.  Falling  objects 

16.  Catapulted  objects 

17.  Heating  devices 

18.  Pumps,  blowers,  fans 

19.  Rotating  machinery 

20.  Actuating  devices 

21.  Nuclear  devices,  etc. 


10.  Moisture 

high  humidity 
low  humidity 

11.  Oxidation 

12.  Pressure 

high  pressure 
low  pressure 
rapid  pressure 
changes 
13-  Radiation 
thermal 

electromagnetic 

ionizing 

ultraviolet 

14.  Chemical  replacement 

15.  Mechanical  shock, 
etc. 


Hazardous  Sources  (89:20) 
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process  extensively  identifies  the  alternative  sequence  of 
events  that  lead  to  a  particular  malfunction  or 
catastrophic  failure,  as  well  as  possible  corrective  action 
for  the  system  in  question,  then  the  assessment  is  called  a 
Preliminary  Hazard  Analysis  (PHA) .  The  PHA  usually 
involves  a  decision  tree  which  aids  in  interrelating  the 
potential  decisions  and  corrective  actions  to  prevent  a 
particular  system  malfunction  or  catastrophic  failure 
(89:21) . 

Failure  Modes  and  Effects  Analysis.  Another  risk 
analysis  technique  is  the  Failure  Mode  and  Effects  Analysis 
( FMEA) .  The  FMEA  is  a  systematic  examination  and  analysis 
of  the  worst-case  impacts  on  the  system  due  to  the  failure 
of  each  component  in  a  system.  As  part  of  the  FMEA,  the 
potential  failure  inodes  of  each  component  are  considered 
and  the  applicable  corrective  action  is  recommended.  Based 
on  the  results  of  the  analysis,  the  components  are  assigned 
Criticality  1,  2  , 3  or  sometimes  4  (89:31). 

Regarding  the  space  shuttle,  NASA  designates  a  system 
component  as  Criticality  1  if  a  single  failure  in  a 
component  can  result  in  "loss  of  life  or  the  vehicle" 
(67:8).  A  component  is  designated  as  Criticality  2  if  a 
single  failure  in  the  component  could  "cause  loss  of 
mission"  (67:8).  Those  components  remaining  after  the 
Criticality  1  and  2  designations  have  been  completed  are 
designated  as  Criticality  3.  However,  NASA  also  uses  a 
category  1R  which  indicates  that  a  particular  item  has  one 
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or  more  backup  (i.e.,  redundant)  item  and  that  failure  of 
the  primary  as  well  as  all  redundant  items  would  cause  loss 
of  life  or  vehicle.  Similarly,  category  2R  indicates  that 
the  primary  as  well  as  all  redundant  components  must  fail 
in  order  to  cause  loss  of  mission  (67:8;  89:31;). 

Fault  Tree  Analysis.  This  technique  involves 
specifying  an  undesirable  event  as  a  top  event  and  then 
constructing  lower  level  failures  that  can  lead  to  the  top- 
level  undesirable  event.  Henley  points  out  that  one 
limitation  of  fault  trees  is  that  as  they  get  very  large, 
"mistakes  are  difficult  to  find,  and  the  logic  becomes 
difficult  to  follow  or  obscured"  (89:48). 

Material  Risk.  The  degree  of  technical  risk  for  a 
development  project  can  often  be  assessed  by  determining 
whether  materials  having  the  required  performance 
characteristics,  and  the  processes  for  forming  these 
materials  are  readily  available  and  have  been  in  relatively 
common  use.  If  present  materials  are  known  to  be  marginal 
for  the  performance  regime  that  is  anticipated  for  the  new 
technology  system,  then  it  is  clear  that  there  will  be 
technical  risk  connected  with  the  selection  and  processing 
of  newer  materials.  For  instance,  aircraft  evolution  has 
been,  and  continues  to  be  very  dependent  on  the  state  of 
materials  technology.  This  is  because  the  greater  the 
speed  at  which  an  aircraft  flies,  tne  higher  is  the  skin 
temperature  of  the  aircraft  as  a  result  of  air  friction 
(140:67,68) .  Increases  in  aircraft  speed  are  often  due  to 
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improved  propulsion  technology.  Such  improvements  usually 
subject  the  engines  to  greater  internal  operating 
temperatures,  thus  height  ending  the  technical  risk.  The 
technical  risks  associated  with  aerospace  vehicles  is 
further  heightened  because  "high  operating  temperatures 
correspond  to  [greater]  fuel  efficiency"  (19:52).  Further, 
the  need  for  high  strength  coupled  with  light  weight  in 
order  to  increase  payload  capacity  (i.e.,  number  of 
passengers,  instrumentation  or  warheads),  may  mean  that  the 
use  of  new  or  exotic  materials  are  mandatory  in  order  to 
meet  system  performance  requirements  (19:51). 

Dr.  Robert  Barthelemy,  manager  of  the  National  Aero- 
Space  Plane  (NASP) ,  presently  in  the  research  phase,  says 
that  the  availability  of  the  appropriate  high  technology 
materials  is  crucial  for  reducing  the  technical  risk 
connected  with  the  NASP  vehicle.  Significant  technical 
risk  arises  from  the  NASP  requirements  to  travel  to  the 
edge  of  space,  and  to  travel  faster  than  Mach  6  in  the 
atmosphere  (133:13). 

Technical  Maturity.  According  to  Major  General  Thomas 
R.  Ferguson,  Deputy  Chief  of  Staff  for  technology  and 
requirements  at  Air  Force  Systems  Command,  it  is  necessary 
to  "  .  .  .ensure  that  the  technology  is  as  mature  as 

possible  in  order  to  reduce  risk  when  we  start  a  new 
program"  (96:7).  Indeed,  the  maturity  of  a  candidate 
technology — which  he  dubs  "technology  readiness"  (96:6)-- 


actually  determines  the  viability  of  development 
alternatives. 

Therefore,  a  technical  risk  assessment  must  include  an 
evaluation  of  whether  materials  with  the  required 
characteristics  are  available,  have  been  tested,  and 
whether  these  candidate  materials  have  been  previously 
applied  with  the  required  degrees  of  success  (19:52,54; 
133:13;  140:68). 

Process  and  Productivity  Risks.  Technical  risk  arises 
also  from  new  processes  used  to  form  materials  into  the 
respective  system  components.  For  instance,  the  now  famous 
Lockheed  "Skunk  Works",  which  built  such  advanced  aircraft 
like  the  Mach  3+  SR-71,  initially  had  significant  problems 
working  with  a  particular  titanium  alloy.  In  spite  of  the 
fact  that  there  were  ten  years  of  research  on  that  titanium 
alloy,  there  had  been  no  use  of  the  alloy  in  a  development 
project  up  to  that  point.  Regarding  the  titanium  alloy, 
Kelly  Johnson,  Skunk  Works  manager,  stated  that  there  was 
".  .  .  a  hell  of  a  gap  between  che  research  and  the 

application"  (8:256).  The  crucial  nature  of  new  processes 
as  sources  of  technical  risk  is  also  underscored  by  the 
initial  difficulty  of  obtaining  semiconductor-grade  silicon 
with  "parts-per-billion  impurity  levels"  (18:9).  Indeed, 
Mayo  has  observed  that  each  advancement  in  electronics, 
communications  and  computers  has  "been  associated  with  new 
materials  or  processing  methods  that  are  more  sophisticated 
than  the  past"  (107:59).  The  foregoing  examples  illustrate 
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that  technical  risk  often  depends  on  whether  or  not  a 
particular  weapon  system  or  component  is  producible.  In 
fact  a  recent  General  Accounting  Office  (GAO)  report  to  the 
Congress  (Reference  COMP)  determined  that  a  smooth 
transition  from  weapon  system  design  to  manufacture  is 
crucial  tor  low  technical  risk.  As  early  as  possible,  the 
risks  attendant  to  a  particular  design  should  be 
identified.  This  requires  that  one  determines 

.  .  .  how  each  part  will  be  made,  and  what  it  will  be 
made  of,  identifying  necessary  production  equipment 
and  skills,  determining  the  layout  and  sizing  of  the 
facility,  and  determining  how  and  when  to  inspect  for 
quality.  (20:2) 

The  selection  of  a  particular  design  is  a  key 
determinant  of  the  degree  of  technical  risk  that  is 
associated  with  production  the  weapon  system  or  component 
in  question.  Accordingly,  design  reviews  must  also  include 
evaluation  of  factors  (20:2,3,9,21,22,27).  In  addition, 
frequent  revisions  to  the  design  of  a  contemplated  weapon 
system  greatly  increase  the  productivity  risk  because  such 
changes  can  invalidate  previous  productivity  planning  and 
preparation.  This  is  especially  the  case  if  the  technology 
involved  is  pushing  the  state-of-the-art  or  if  the  design 
changes  are  occurring  relatively  late  (20:31-33).  All 
design  changes  therefore  drive  updates  and  reevaluation  of 
previous  analyses  (28:66). 

Quality .  However,  the  technical  risk  arises  not  just 
from  whether  or  not  a  particular  design  is  producible,  but 
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also  from  whether  or  not  the  item  can  be  produced  with  the 
desired  level  of  quality  that  is  required  or  requested. 

For  example 


.  .  .  missiles  are  complex  systems  with  numerous 

components  and  parts  required  in  their  assemblies. 

Each  requires  production  processes  and  many  thousands 
off  individuals  using  an  arr  y  of  production  and 
testing  equipment,  to  manufacture  and  deliver  a 
quality  product.  The  opportunities  for  defects  to 
occur  are  immense  and  one  defect  can  render  a  missile 
ineffective.  (66:4) 

Specifically,  defective  material  and  workmanship  during  the 
building  of  a  system  can  contribute  markedly  to  the 
technical  risk.  A  recent  General  Accounting  Office  (GAO) 
report  states  that  "soldering  defects  can  and  have  lead  to 
missile  failures"  (66:28).  Further  technical  risk  arises 
from  the  long  term  deleterious  environmental  effects  on  the 
solder  joints.  This  is  of  special  concern  since  some 
missiles,  such  as  the  Phoenix  missile,  have  over  60,000 
solder  joints  (66:28-30). 

Prerequisite  Technologies.  An  adequate  technical  risk 
assessment  must  also  take  into  account  whether  certain 
prerequisite  technologies  required  for  the  success  of  the 
project  in  question  are  sufficiently  well  developed.  For 
instance,  early  computers  were  built  using  vacuum  tubes. 
However,  more  significant  computer  advances  were  achieved 
only  with  the  advent  of  the  transistor  (8:253,266,257; 
18:6). 
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Human-Induced  Malfunctions 


A  GAO  study  emphasized  that  human  factors  and 
reliability  concerns  are  often  inadequately  considered 
during  the  design  of  a  system.  In  fact,  some  studies  have 
found  that  "human  errors  account  for  at  least  50  percent  of 
the  failures  of  major  systems"  (54:27).  Those  errors 
represent  a  major  source  of  risk  to  proper  system 
performance.  Because  the  human  is  often  a  component  of  the 
system  performance,  the  reliability  of  the  human  should  be 
considered  as  a  component  of  the  system  technical  risk. 
However  a  mission  or  system  failure  may  be  caused  by  a 
human  performing  a  procedure  incorrectly  or  selecting  the 
wrong  procedure  although  this  would  not  necessarily  lead  to 
equipment  failures  (54:28-30).  Human  errors  fall  into  the 
five  categories  of 

1.  Failure  to  follow  procedures 

2.  Incorrect  diagnosis  of  a  particular  situation 

3.  Misinterpretation  of  communication  (Written  or 
verbal ) 

4.  Inadequate  support,  tools,  equipment  and 
environment 

5.  Insufficient  attention  or  caution.  (54:30) 

For  correct  system  design  the  designer  must  take  into 
account  certain  key  characteristics  of  the  personnel  that 
will  be  operating  and  maintaining  the  system.  Some  of 
these  key  human  factors  are 

1.  Muscular  strength  and  coordination 

2.  Body  dimension 

3.  Perception  and  judgement 

4.  Sensory  capacities 
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5.  Native  skills  and  capacity  to  learn  new  skills 

6.  Optimum  workload 

7.  Basic  requirement  for  comfort,  safety,  and  freedom 
from  environmental  stress.  (54:29) 

Quantitative  Techniques  of  Risk  Assessment 

As  stated  above,  the  opinion  of  experts  is  often  a 
critical  component  of  a  qualitative  risk  assessment.  Such 
qualitative  assessments  are  often  a  prerequisite  for 
follow-on  quantitative  risk  assessments  (24:11-1;  89:24; 
132:40).  The  major  techniques  in  use  today  for 
quantitatively  assessing  project  risk  are: 

1.  network  analysis, 

2.  the  method  of  moments, 

3.  decision  analysis, 

4.  Work  breakdown  system  simulation, 

5.  Graphics, 

6.  Estimating  relationships 

7.  Risk  factors.  (24:Gec  IV,i) 

Although  these  techniques  focus  largely  on  cost  and 
schedule  risk,  a  technical  assessment  of  some  sort  is 
normally  a  component  of  each  of  these  techniques. 

Therefore,  these  techniques  will  be  examined  here.  Since 
the  techniques  are  discussed  in  some  detail  in  Risk 
Assessment  Techniques;  A  Handbook  for  Program  Management 
Personnel  (24)  only  a  summary  description  of  each  will  be 
given  here. 

Network  Analysis.  Network  analysis  gained  renown 
because  of  its  use  in  the  development  of  the  Polaris 
submarine.  A  network  is  composed  of  a  series  of  lines  and 
circles  which  are  present,  respectively,  the  activities  and 
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the  important  intermediate  objectives  which  will 
collectively  result  in  the  successful  achievement  of  the 
project.  In  particular,  such  activity  lines  and  objective 
circles  must  reflect  the  interrelationships  and  the 
sequence  that  is  required  for  project  success  (24:IV-2-5). 
The  Program  Evaluation  and  Review  Technique  (PERT)  is  a 
network  technique  that  was  primarily  used  as  a  tool  for 
managing  project  schedule.  Time  durations  for  completing 
each  activity,  and  the  respective  probabilities  for  on-time 
completion  of  each  of  the  activity  are  assigned  to 
determine  which  objective  is  at  risk  of  being  achieved 
late,  and  to  assess  the  schedule  risk  posed  to  subsequent 
or  interrelated  objectives.  Computer  simulation  can  then 
be  used  to  evaluate  the  pessimistic,  most  likely,  and 
optimistic  completion  times  for  the  entire  project 
(124:216-221) . 

Method  of  Moments.  The  method  of  moments  is  a  program 
risk  assessment  method  which  is  primarily  concerned  with 
cost  risk.  This  method,  unlike  the  network  method  just 
described,  does  not  use  a  sequence  of  network  relations. 
Instead,  the  collective  effect  of  the  probability 
distribution  functions  associated  with  program  elements  in 
a  Work  Breakdown  Structure,  are  determined  mathematically 
instead  of  through  simulation  (24 : IV-10, IV-11) . 

Decision  Analysis.  Decision  analysis  is  a  risk 
assessment  method  whereby  decisions  are  broken  down  into 
sequences  of  constituent  and  supporting  decisions.  Such 
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decisions  (i.e.,  outcomes)  are  usually  represented  by  a 
"tree”  where  the  branches  from  each  node  represent  possible 
outcomes.  Probabilities  of  each  of  the.  outcomes  are 
usually  assigned  and  expected  values  for  each  outcome  can 
then  be  calculated.  Specific  variations  and  enhancements 
allow  decision  trees  to  be  used  for  calculating  and 
comparing  the  expected  values  for  a  number  of  different 
decision  sequences  regarding  cost  and  schedule  (24:IV-7). 

WASH  1400 — the  landmark  risk  assessment  study 
mentioned  previously — used  variations  of  decision  trees  to 
assess  the  risk  of  accident  whereby  a  nuclear  reactor  would 
release  toxic  radioactive  fission  products  (89:24). 

Work  Breakdown  Simulation.  The  work  breakdown 
simulation  method  is  essentially  the  same  as  the  method  of 
moments  except  that  computer  simulation  is  used  to  obtain  a 
probability  density  function  (24:IV-13). 

Graphic  Method.  The  graphic  method  is  another  method 
that  is  used  to  assess  the  cost  risks  of  a  particular 
program.  In  this  technique,  the  cumulative  distribution 
functions  (CDF)  of  the  individual  cost  elements  (cost 
versus  the  probability  of  that  particular  cost)  is  combined 
to  form  a  collective  CDF  for  the  overall  program  (24: IV- 
14,15, 16,17) . 

Estimating  Relationship  Method.  The  estimating 
relationship  method  is  a  technique  that  uses  appropriate 
historical  cost  data  to  determine  the  amount  of  contingency 
funding  that  should  be  reserved  to  cover  the  unanticipated 
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development  expense  due  to  the  technological  risk  involved. 
This  technique  requires  that  factors  such  as  "engineering 
complexity,  degree  of  system  definition,  contractor 
proficiency/experience,  and  multiple  users"  (24:19)  be 
evaluated  (24:18,19). 

Risk  Factor  Method.  The  rif.k  factor  method  uses  a 
technical  Work  Breakdown  Structure  that  assigns  relative 
weighting  corresponding  to  the  degree  of  anticipated 
technical  risk  associated  with  the  system  components.  The 
weighting  allows  costs  estimates  for  development  costs  to 
be  assigned  commensurate  with  technical  risk,  thereby 
minimizing  the  risks  that  the  estimated  costs  are  not 
realistic  (24:1V— 21).  There  are  a  number  of  computer 
programs  presently  in  use  in  which  the  appropriate  input 
information  results  in  an  output  which  identifies  the  cost 
risk  to  a  project.  For  instance  the  Army  has  developed  the 
Total  Risk  Assessing  Cost  Estimates  (TRACE)  software  to 
identify  the  cost  risks  of  projects  in  an  entire  command 
and  to  aid  in  assigning  funds  commensurate  with  the  cost 
risks  (24:23;  30:Sec  15,13). 

Design  Comparisons.  One  method  of  assessing  technical 
risk  is  by  comparing  the  design  of  an  existing  or  proposed 
system  to  the  design  of  a  system  that  has  had  a  performance 
failure.  Such  a  comparison  may  be  quantitative  qualitative 
or  both.  This  technique  has  been  used  to  determine  the 
accident  risk  of  a  number  of  United  States  nuclear  reactors 
by  comparing  them  to  the  reactor  which  had  an  accident  near 


Chernobyl  in  the  Soviet  Union.  Refer  to  the  case  study  on 
the  Chernobyl  Nuclear  Reactor  Accident  for  an  example  of 
how  design  comparisons  are  made  (62:1-6;  63:1-6).  A 
comparison  between  an  existing  and  a  proposed  design  or 
system  is  also  valuable  for  ascertaining  that  the  time 
allotted  for  achieving  the  proposed  design  or  system  is 
realistic.  An  unrealistically  short  time  could  increase 
technical  risk  since  some  recommended  technical  tasks  could 
be  abridged  or  eliminated.  For  example,  a  GAO  report  cited 
that  the  time  scheduled  for  the  development  of  a 
particular  trainer  aircraft  was  ‘'probably  too  short  given 
the  history  of  problems  with  engine  development  programs*' 
(75:16).  Because  time  is  often  a  factor  in  technical  risk 
reduction,  technical  productivity  aids  should  be  identified 
and  used.  One  important  class  of  such  productivity  aids 
are  design  tools  which  "cover  the  spectrum  from  computer 
simulations  supporting  design,  to  computer-integrated 
manufacturing"  (29:32).  These  tools  have  been  cited  for 
increasing  the  "capability  of  engineers  [by]  300-3500%" 
(29:84) . 

Risk  Management 

In  general,  any  action  which  reduces  the  uncertainties 
or  unknowns  associated  with  achieving  a  desired  outcome  is 
in  essence  a  risk  management  technique.  Accordingly,  a 
risk  assessment  "is  the  first  step  in  risk  management" 
(30:Sec  1,2).  A  study  sponsored  by  the  Defense  Advanced 
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Research  Projects  Agency  (DARPA)  defined  risk  management, 
as  it  regards  a  project,  as  a  process  of  evaluating, 
decreasing  or  avoiding  the  uncertainties  associated  with 
meeting  objectives  so  that  a  program  will  successfully 
achieve  its  goals  (113:1). 

Techniques  of  Technical  Risk  Management 

Ideally,  the  best  method  of  managing  technical  risk  is 
to  eliminate  risk  by  use  of  an  appropriate  design.  This  is 
especially  preferred  where  safety  risk  (i.e.,  a  hazard)  is 
concerned.  However,  rarely  will  all  risk  be  eliminated, 
therefore  some  technical  risk  management  will  have  to  be 
pursued  (3:113;  36:6).  Incidentally,  it  is  highly 
important  that  the  key  technical  criteria  and  parameters 
are  identified  for  a  system  undergoing  development  that  can 
be  used  as  indicators  of  system  technical  health,  progress 
or  lack  of  progress  (25:Sec  8,6).  The  following  general 
types  of  technical  risk  reduction  techniques  have  been 
cited: 

1.  Test  and  Evaluation 

2.  Use  of  Prototypes  and  Demonstration 

3.  Studies  and  Analyses 

4.  Development  of  New  Ideas  and  concepts.  (143:21) 

The  Air  Force  MIL-STD-152 IB  cites  the  following 
technical  risk  reduction  techniques: 

1.  Adequate  trade-offs  (particularly  for  sensitive 
mission  requirements  versus  engineering  realism 
and  manufacturing  feasibility  to  satisfy  the 
production  capabilities) ; 
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2.  Subsystem/component  hardware; 

3.  A  responsive  test  program; 

4.  Implementation  of  comprehensive  engineering 
disciplines  (e.g.,  worst  case  analysis,  failure 
mode  and  effects  analysis  producibility  analysis 
and  standardization).  (28:26) 

Risk  reduction  may  also  be  achieved  by  selectively 
applying  military  specifications  and  standards,  when 
appropriate,  since  they  are  a  "valuable  corporate  memory 
(based  on]  .  .  .  past  experience" ( 2 5 : Sec  8,8). 

Design .  A  fundamental  guideline  of  risk  management  by 
design  is  sim  Lification.  That  is,  the  design  of  the 
system  should  be  "no  more  complex  than  necessary  to  perform 
the  function  reliably"  (29:39).  By  minimizing  the  number 
of  parts  and  interconnections,  simplifying  the  support 
tools  and  equipment,  and  providing  easy  access  to  test 
points,  the  system  will  be  "simpler  to  build,  operate, 
support  and  upgrade"  (29:40).  Modularity  simplifies  the 
system  design,  troubleshooting,  and  software  and  hardware 
revision.  Software  modularity  also  reduces  the  likelihood 
of  logic  errors  and  aids  troubleshooting  (29:44,45).  For 
maximum  benefit,  analyses  such  as  Fault  Tree  Analyses  and 
Failure  Modes  and  Effects  Analyses,  should  be  done  early 
enough  so  that  the  results  can  be  used  "to  influence  the 
design,  not  just  to  document  it  [the  design]"  (29:67). 

Since  analyses  may  indicate  system  deficiencies  and 
inadequate  design  margins,  the  timing  of  analyses  are  very 
important  for  technical  risk  reduction  (29:65-67).  One 
basic  risk  reduction  technique  is  to  incorporate  design 
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margins  whereby  parts  are  selected  which  are  rated  to 
withstand  much  higher  loads  or  capacity,  as  applicable, 
then  they  are  expected  to  encounter  when  used  in  the 
system.  This  practice  in  electronic  design  is  called 
derating,  and  results  in  increased  reliability  (i.e., 
decreased  failure  rate)  since  the  parts  "experience  stress 
levels  below  their  rated  values"  (29:78-79). 

Testing .  Testing  has  been  identified  as  so  crucial  to 
the  technical  risk  reduction  process  that  the  Department  of 
Defense  position  is  that:  "testing  will  begin  as  soon  as 
possible  ...  to  reduce  acquisition  risk  and  to  estimate 
the  capability  of  the  system  under  development"  (12 : 2)  . 

For  example,  testing  revealed  that  the  M-l  tank  met 
such  combat  requirements  as  firepower  and  mobility,  but 
that  the  tank's  power  train  did  not  meet  the  Army's 
durability  goal  (59:ii).  Similarly,  laboratory  tests  of 
the  Army's  viper — a  light  anti-tank  weapon  fired  from  the 
shoulder  of  an  infantryman — revealed  that  "static 
electricity  or  radar  waves  may  cause  the  Viper  to 
accidentally  fire"  (58:48).  Operational  tests  revealed 
that"  20  of  368  Viper  rounds  failed  to  fire  on  the  first 
attempt  .  .  .  (due  to]  poor  quality  .  .  .  defective 
batteries  .  .  .  gunner  error  .  .  .  (or)  unknown  [causes]" 
(58:49).  For  effective  testing  to  be  performed,  detailed 
test  planning  must  be  completed  so  that  the  necessary  test 
resources  can  be  identified  or  developed,  and  the  critical 
test  issues  can  be  ascertained  (52:ii). 
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It  is  important  that  the  testing  of  a  system  or 
component  be  very  realistic.  Equipment  may  function  well 
at  ambient  temperature  but  may  fail  to  function  correctly 
when  subjected  to  the  extremely  cold  or  hot  temperatures 
that  the  equipment  is  likely  to  encounter.  For  example, 
the  Air  Launched  Cruise  Missiles'  (ALCM)  Flight  Data 
Transmitters  (FDT)  were  sensitive  to  cold  temperatures.  In 
two  different  operational  launch  attempts,  an  FDT 
malfunctioned  during  the  pre-launch  test  due  to  the  effects 
of  cold  temperature.  Both  FDT's  appeared  to  function 
properly  when  tested  on  the  ground  at  ambient  temperatures, 
but,  when  later  chilled  in  a  laboratory,  both  failed 
(73:13).  The  sensitivity  of  FDT's  to  cold  temperatures  was 
especially  important  since  "SAC  bombers,  which  launch  the 
ALCM,  flying  at  the  32,000-52,000  feet  required  for 
strategic  missions,  could  encounter  temperatures  as  low  as 
below  [minus]  116  degrees  F"  (73:13). 

A  fundamental  example  of  the  importance  of  test 
realism  in  reducing  technical  risk  is  illustrated  by  the 
case  of  the  Army's  High  Mobility  Multipurpose  Wheeled 
Vehicle  (HMMWV) .  Although  the  HMMWV  performed  well  during 
development  testing,  several  deficiencies  became  evident  as 
a  result  of  the  more  realistic  operational  testing.  During 
the  operational  tests  it  was  discovered  thrt  the  vehicle's 
radiator  "was  subject  to  clogging  in  dusty  and  sandy 
environments"  (51:5).  Further,  the  HMMWV's  air  induction 
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system  allowed  "dirt  and  water  to  enter  the  engine"  (51:5). 
This  was  a  serious  deficiency  since  the  vehicle  would  be 
vulnerable  to  mud,  dust  and  water  during  cross-country 
maneuvers.  To  resolve  the  stated  deficiencies,  and  to 
thereby  reduce  the  technical  risks,  corrective  design 
modifications  were  planned  for  the  HMMWV  prototype  (51:5- 
8)  . 

The  operational  testing  is  normally  conducted  by  an 
independent  organization  because  one  cannot  "expect  a  man 
or  organization  who  created  a  system  to  discover  its 
faults"  (125:6).  However,  the.  performance  of  any  test  does 
not  actually  eliminate  technical  risk.  Only  the 
implementation  of  corrective  actions  or  design 
modifications  that  are  shown  to  be  necessary  by  the 
testing,  will  actually  reduce  technical  risk  (82:258-268). 

Simulation  and  Analysis.  Since  a  test  article  may  not 
be  available  at  the  inception  of  a  new  technology 
development  program,  there  should  be  heavy  emphasis  on 
simulation,  modeling  and  analysis  to  begin  to  assess  the 
technical  risks  and  limitations  of  prospective  design 
approaches  (32:7)  . 

However,  the  deficiencies  in  a  simulation  may  result 
in  the  introduction  of  additional  technical  risk.  For 
example,  the  GAO  found  that  "certain  target  and  simulation 
shortcomings"  (71:3)  meant  that  some  specific  Anti- 
Submarine  Warfare  simulations,  being  performed  by  the  Navy 
o  test  the  Advanced  Capability  and  MK— 50  torpedoes, 
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1.  Is  (he  system  operated  by  typical  units? 

2.  Operated  by  typical  operating  personnel? 

3.  Supported  by  typical  support  units? 

4.  Equipment  put  under  realistic  stress  by  design? 

(e  g.  outer  envelope  ol  performance  requirements  tested?) 

5.  Personnel  put  under  realistic  stress  by  design? 

6.  Realistic  combat  tactics? 

7.  Oid  the  physical  environments  approximate  the  intended  range  of 
environments  (terrain,  time  of  day,  weather,  tea  State,  clutter. 

Iff)? 

8.  Oid  target  systems  approximate  actual  targets,  realistically 
employed? 

9.  Oid  threat  systems  approximate  actual  threat,  realistically 
employed? 

10  Was  the  tested  system  production  representative  and  prepared  for 
the  test  in  a  realistic  manner? 


Figure  5.  Framework  for  Evaluating  the  Effectiveness; 
of  Operational  Test  and  Evaluation  (76:34-35) 
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incompletely  represented  "important  environmental  factors 
.  .  .  and  may  increase  technical  risks  of  weapons  not 

performing  as  intended"  (71:3).  Therefore,  effective 
evaluations  of  test  resources  are  necessary  to  determine 
the  utility  and  validity  of  the  test  results  that  are 
obtained  using  a  particular  test  resource  (71:4) . 

In  particular,  certain  weapon  systems  must  be  tested 
in  the  electronic  warfare  environment  since  their 
performance  may  be  markedly  degraded  in  such  an 
environment.  Simulated  threat  environments  which  are 
inaccurate  could  introduce  technical  risk  (52:2).  For 
example , 


.  .  .  the  Air  Force's  Sparrow  air-to-air  missile  was 

tested  against  aerial  targets  that  did  not 
realistically  represent  the  actual  threat.  When  first 
used  in  Vietnam,  the  Sparrow  missile  missed  its  target 
more  than  it  hit.  (52:6) 


A  relatively  new  technical  and  test  concern  is  whether  or 
not  the  crucial  control  electronics  of  a  particular  system 
can  continue  to  function  when  subjected  to  the 
electromagnetic  pulse  radiation  from  nuclear  explosions  in 
air  or  space  (57:21;  110:164). 

Goodman  points  out  that  it  is  prudent  to  be  aware  that 
one  or  more  technical  approaches  may  look  good  on  paper  but 
must  be  tested  to  ascertain  that  these  contemplated 
approaches  can  be  reduced  to  practice.  In  some  cases,  he 
states,  analyses  are  inadvisable  for  assessing  the 
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technical  risk  and  some  sort  of  test  must  be  implemented  to 
gauge  technical  risk  (38:A-28;  58:258-261). 

Detailed  Failure  Analyses.  The  availability  of 
facilities  to  conduct  failure  analyses  on  subsystems  or 
components  of  prototype  hardware  can  also  contribute 
immensely  toward  risk  reduction  since  the  lessons-learned 
can  be  used  to  improve  the  technical  design.  For  instance 
the  Rome  Air  Development  Center  has  a  Quick  Reaction 
Failure  Analysis  Laboratory  that  can  conduct  detailed 
analysis  of  failures  in  semiconductor  devices.  Based  on 
the  results  of  failure  analyses,  appropriate  design  changes 
can  be  made  by  the  DOD  program  in  question.  Therefore  the 
reduction  in  technical  risk  that  may  be  derived  from 
labora-tory  support  can  include  the  use  of  sophisticated 
laboratory  instruments  (e.g.,  scanning  electron  microscope, 
pattern  generators,  oscilloscopes,  infrared  scanners)  and 
non— destructive  tests  (e.g.,  X-rays)  for  electronic 
microcircuit  failure  analysis,  as  well  as  technical 
assessments  of  contemplated  design  approaches  (40:42,43). 

Failure  analysis  personnel  may  also  be  employed  to 
prevent  certain  failures.  For  example,  the  growth  of  "tin 
whiskers"  within  some  electronic  components  is  an  incipient 
failure  mode  that  can  ultimately  result  in  internal 
shorting  in  hybrid  and  integrated  circuits.  This  means 
that  weapon  systems  that  have  tested  functional  before 
storage  could  potentially  fail  those  same  tests  alter 
storage.  Fortunately,  the  Department  of  Defense  has  been 
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alerted  of  this  particular  failure  mode  and  can  minimize 
the  risk  by  selecting  proper  materials.  Refer  to  Appendix 
C  for  detailed  correspondence  on  the  "tin  whisker"  failure 
mode.  Similarly,  Appendix  I  outlines  the  risk  of 
electronic  component  damage  due  to  electrostatic  discharge. 

Software  Testing.  Thorough  testing  of  software,  as 
part  of  software  verification  and  validation,  has  been 
established  in  numerous  technical  programs  as  indispensable 
for  precluding  low  software  reliability  and  to  verify  that 
user  requirements  have  been  achieved  by  a  system  in 
question.  A  GAO  report  cited  that  the  development  of 
application  software  is  such  a  laborious  "and  error  prone 
process,  and  errors  can  be  made  both  in  deciding  what  the 
program  should  do,  and  in  writing  them  to  do  it"  (56:1). 

To  enhance  the  testing  objectivity  and  to  increase  the 
chances  that  user  requirements  were  correctly  understood 
and  implemented  in  the  software,  independent  testing  and 
evaluation  of  the  software  should  be  conducted  in  addition 
to  in-house  testing  (56:38,39).  Due  to  the  labor  intensive 
nature  of  software,  test  tools — such  as  test  data 
generators — should  be  used  both  to  increase  the  software 
testing  effectiveness  and  the  productivity  of  the  software 
effort.  However,  manual  techniques  can  be  used  to 
supplement  the  ?utomated  testing  techniques.  Further, 
adequate  personnel  and  time  should  be  allotted  for  the 
software  effort  (56:10,12). 
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Contingency  Planning.  Another  method  of  minimizing 
risk  is  to  plan  for  contingencies.  Thiu  involves  looking 
at  possible  "what-if"  situations  and  laying  out  possible 
courses  of  action  before  the  particular  situation  occurs. 
This  contingency  planning  may  be  taken  further  by 
evaluating  the  options  corresponding  to  certain  worst-case 
scenarios  (28:24;  98:280).  Goodman  states  that  as  a 
contingency  measure,  it  is  highly  important  to  identify  and 
carry  along  (i.e.,  demonstrate)  at  least  one  lower 
technology  approach  corresponding  to  each  high  technology 
approach  in  order  to  have  a  fallback  position  if  the  high 
technology  approach  is  not  achieved  (79:262). 

Such  contingency  planning  should  include  a 
consideration  of  relatively  mundane  sources  of  technical 
risk.  For  example,  the  21  July  1961  flight  of  the  Mercury 
capsule  into  space  made  Captain  Virgil  Grissom  the  second 
American  in  space.  There  had  been  concern  that  upon 
splashdown  in  the  ocean,  the  space  capsule  would  become 
submerged  (instead  of  floating  on  the  surface).  To  allow 
the  astronaut  to  escape,  an  innovative  explosive  cord  had 
been  placed  around  the  hatch.  However  upon  splashdown,  the 
hatch  apparently  blew  inadvertently,  causing  sea  water  to 
enter  the  capsule.  To  avoid  sinking  with  the  capsule, 
Grissom  quickly  scrambled  out  of  the  open  hatch.  Grissom's 
was  designed  to  allow  him  to  float  on  water,  by  closing  off 
a  "neck  dam"  (23:XVII).  However,  in  quickly  leaving  the 
sinking  capsule,  Grissom  forgot  to  close  off  "an  intake 
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port  in  the  center  of  the  suit  [and]  .  .  .  water  began  to 
enter  the  suit*'  (23:XVII).  Having  endured  the  6  g  force  of 
ascent  and  the  10.2  g  forces  of  re-entry,  it  was  clear 
according  to  one  doctor  that  Grissom  came  close  to  drowning 
during  the  splashdown  (23 :XVII , XVIII) . 

Parallel  Effort.  To  reduce  the  technical  risk, 
"simultaneous  parallel  efforts  are  encouraged  [early  in  a 
DOD  Acquisition  process)  ...  to  reduce  the  risk  from 
uncertain  technology  and  development"  (143:15).  Such 
parallel  design  efforts  pursue  the  same  technical 
objectives  in  order  that  a  number  of  different  design 
approaches  are  available  to  be  evaluated  for  superiority 
(143:15) . 

For  example,  on  4  October  1957  the  Soviet  Union  was 
the  first  to  put  a  man-made  satellite — Sputnik — into  orbit. 
The  United  states'  first  attempt  to  achieve  the  same  feat 
ended  when  the  Vanguard  rocket  exploded  on  the  launch  pad 
during  the  attempted  launch  on  6  December  1957.  However, 
the  United  States  Army  had  been  building  the  Explorer  I 
vehicle  which  was  successfully  launched  on  31  January  1958. 
The  nearly  simultaneous  availability  of  both  the  Vanguard 
and  the  Explorer  I  launch  vehicles  provided  alternatives 
for  achieving  the  United  States  launch  objective 
(86: 148, 160, 162) . 

Incremental  Development.  The  use  of  an  incremental 
development  strategy  can  greatly  reduce  technical  risk. 
Under  this  approach  only  established  technology  having 
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little  development  risk  would  be  implemented  in  an  ongoing 
program  and  each  successive  version  of  the  hardware  could 
include  newly  matured,  proven  technologies  and 
enhancements.  A  variation  of  this  approach  is  called  Pre- 
Planned  Product  Improvement,  whereby  the  option  to  upgrade 
the  completed  or  deployed  system  to  higher  technology  would 
have  been  preserved  all  along  by  appropriate  design  and 
planning.  An  example  of  the  incremental  development 
approach  was  the  United  States  Gemini  space  vehicle  series. 
The  Gemini  space  vehicle  "drew  heavily  on  proven  Mercury 
[predecessor  program]  technology"  (92:65),  therefore  the 
Gemini  space  vehicle  were  much  more  advanced  and  versatile 
(132:33) . 

Technical  Enrichment.  Goodman  has  stated  that  one  of 
the  techniques  of  minimizing  technological  risks  is  to 
ensure  that  the  technical,  human  resources  involved  with 
the  project  are  continually  being  enriched  by  innovative 
ideas  and  perspectives.  To  achieve  the  said  enrichment, 
new  technical  personnel  can  be  recruited  into  the 
organization,  attendance  at  professional  technical  meetings 
can  be  encouraged,  and  an  appropriate  emphasis  on  training 
can  be  employed.  However,  Goodman  repeatedly  emphasizes 
that  for  risk  reduction,  "the  key  management  task  is  to 
create  an  early  learning  path"  (82:258).  Especially  if  the 
staff  involved  is  new  or  inexperienced,  Goodman  stresses 
that  it  is  necessary  to  "rush-to-prototype"  (82:262)  to 
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provide  the  staff  with  experience  that  is  specific  to  the 
technical  endeavor. 

Case  Studies 

A  number  of  case  studies  of  technical  risk  management 
have  been  selected  based  on  the  criteria  specified  in 
Chapter  II.  These  case  studies  reveal  significant  lessons 
and  potential  pitfalls  in  the  management  of  technical  risk. 
Since  the  case  studies  encompass  past,  contemporary  and 
prospective  systems,  they  provide  a  wide  vista  for  the 
examination  of  technical  risk  management  in  action. 

The  F/A-18  Strike  Fighter.  Designed  for  Navy  and 
Marine  Corps  use,  has  two  engines,  "redundant  flight 
control  computers  and  a  backup  mechanical  flight  control 
system"  (55:1) .  Such  redundancies,  besides  enhancing 
reliability,  are  also  intended  to  enhance  the  survivability 
of  the  aircraft  during  a  combat  mission.  The  software 
development  for  the  F/A-18  flight  control  system  was  behind 
schedule  because  Navy  and  contractor  personnel 
underestimated  the  amount  of  work  necessary.  Also, 
requirement  changes,  driven  by  the  need  to  correct  a 
roll— rate  problem,  adversely  affected  the  software  effort. 
Further,  "almost  all  of  the  flight  control  computer's 
memory  space  had  been  used  while  demands  for  additional 
space  continued"  (55:8) . 

The  use  of  composite  materials  on  some  F/A-18 
surfaces,  instead  of  aluminum  and  titanium,  resulted  in 
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added  technical  risk.  Specifically,  poor  predictions  were 
made  regarding  the  flexibility  of  composite  structures 
since  the  aircraft  industry  was  less  familiar  with 
composite  structures  than  with  aluminum  and  titanium  (55:8- 
11) .  To  enhance  maintenance,  *  built-in  test  system  was 
being  developed  for  the  F/A-18,  however,  as  of  1980  a 
relatively  high  number  of  built-in  test  false  alarms  had 
been  seen  (55:22). 

The  F/A-18  development  and  test  effort  was  not  without 
catastrophes  because  ".  .  .  in  September  1900  a  development 
F/A-18  aircraft  crashed  in  England  because  of  a  failure  in 
the  low  pressure  turbine  [disk]  of  one  of  its  F404  engines" 
(55:ii).  Also,  "on  November  14,  1980,  during  an 
operational  test  and  evaluation  .  .  .  [an  F/A-18]  aircraft 
entered  into  a  spin  while  practicing  combat  maneuvers,  and 
the  pilot  was  unable  to  regain  control"  (55:ii). 

Chernobyl  Nuclear  Reactor  Accident.  In  early  1968, 
the  reactor  core  of  a  "graphitemoderated,  water-cooled 
nuclear  reactor"  (63:2)  exploded  near  Chernobyl  in  the 
Soviet  Union  killing  31  people  and  spreading  deadly 
radiation.  The  Chernobyl  reactor  accident  was  attributed 
to  "a  combination  of  human  error  and  poor  reactor  design" 
(63:8).  To  assess  the  risk  that  some  United  States 
reactors  could  fail  the  same  way,  some  comparisons  were 
made  between  some  United  States  reactors  and  the  Chernobyl 
reactor.  In  particular,  a  GAO  study  found  that  the 
Chernobyl  reactor  and  the  Fort  St.  Vrain  (United  States) 
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reactor  both  "a  graphite  core  to  control  the  rate  of 
fission  within  the  reactor”  (63:2).  However,  the  Chernobyl 
design  allowed  only  seconds  of  response  time  to  avoid  the 
equivalent  accident.  This  difference  in  allowable  response 
time  is  due  to  the  difference  in  the  cooling  system  design 
in  the  two  reactors  (63:2,3,33;  83:39-42).  The  Fort  St. 
Vrain  reactor  was  specifically  "designed  so  that  the 
graphite  in  the  reactor  core  absorbs  most  of  the  heat” 
(63:3).  That  heat  is  then  transferred  to  helium  gas  that 
circulates  through  the  graphite.  However,  in  the  Chernobyl 
reactor,  the  graphite  was  specifically  designed  only  for 
fission  moderation  and  not  especially  for  heat  absorption. 
Instead,  the  Chernobyl  reactor  uses  cooling  water  to  absorb 
the  heat.  Therefore,  "the  consequences  of  a  loss-of- 
coolant  accident  would  be  more  severe  at  a  Chernobyl-type 
reactor  than  at  Fort  St.  Vrain"  (63:3).  According  to  a 
Soviet  report,  the  Chernobyl  accident  occurred  while  the 
reactor  operators  were  conducting  a  test  which  required 
that  a  number  of  reactor  safety  systems  be  disabled 
(63:32).  However,  the  Chernobyl  reactor 

.  .  .  was  not  designed  to  withstand  loss  of  coolant 

without  the  use  of  the  emergency  cooling  system.  If 
the  cooling  is  inadequate  for  a  very  short  period 
(seconds)  .  .  .  the  fuel  would  begin  to  melt.  This 

would  result  in  over-pressurization  of  the  fuel  tubes 
and  explosions.  (63:21) 

It  is  very  clear  that  the  hazardous  events  happend 
with  such  rapidity  that  past  a  certain  point  the  Soviet 
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technicians  could  not  respond  quickly  enough  because  "in 
less  than  20  seconds,  the  reactor  power  level  increased  to 
100  times  its  designated  power  level"  (63:32).  In  fact, 
"the  power  surge  was  so  rapid  that  neither  the  control  rods 
nor  the  coolant  could  be  adjusted  fast  enough  to  stop  the 
accident"  (63:33). 

Airplane  Development .  On  17  December  1903  at  about 
10:35  in  the  morning  near  K’tty  Hawk,  North  Carolina, 
Orville  Wright — with  his  brother  Wilbur  and  five  witnesses 
on  the  ground — successfully  flew  the  first  airplane. 

However  Professor  John  D.  Anderson,  chairman  of  the 
Department  of  Aerospace  Engineering  at  the  University  of 
Maryland,  states  that 

Contrary  to  some  popular  belief,  the  Wright  Brothers 
did  not  invent  the  airplane?  rather  they  represent  the 
fruition  of  a  century  of  prior  aeronautical  research 
and  development.  The  time  was  ripe  for  the  attainment 
of  powered  flight  .  .  .  The  Wright  Brothers'  ingenuity 

dedication  and  persistence  .  .  .  [made  them]  first. 
(5:3) 

The  efforts  of  the  British  inventor  Sir  George  Cayley 
significantly  advanced  the  theoretical  understanding  of 
aeronautics  and  was  therefore  instrumental  in  reducing  some 
of  the  technological  risk  of  attaining  the  1903  flight  with 
the  Wright  Brothers'  aircraft.  In  1804  Cayley  built  a 
revolving-arm  mechanism  which  he  used  for  experimenting 
with  airfoils.  Also,  in  1804,  Cayley  built  and  flew  a 
small  (one-meter  long)  model  glider.  In  1849  Cayley  built 
and  flew  a  full  scale  glider  that  had  three  wings  mounted 
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above  each  other.  On  a  number  of  occasions,  a  small  boy  on 
the  glider  was  lifted  several  meters  into  the  air  when  the 
glider  was  traveling  down  a  hill  (5:8,9,10;  47:219; 

109:397) . 

Other  aeronautical  experiments,  such  as  the  more  than 
2,500  successful  flights  that  Otto  Lillienthal  made  in  his 
hang  gliders,  clearly  reduced  the  technical  risk.  On  a  9 
August  1896  flight,  Lillienthal  crashed,  broke  his  back, 
and  died  the  next  day.  His  last  words  were  (translated  from 
German):  "Sacrifices  must  be  made"  (142:43).  The  Wright 

Brothers  followed  the  Lillienthal  hang  glider  flight 
experiments  intently  through  written  accounts  and  some  of 
Lillienthal ' s  reports  (5:18).  It  was  Lillienthal  who  first 
showed  that  curved  wing  surfaces  were  superior  to  flat  wing 
surfaces  for  gliding  (43:242) .  Wilbur  Wright  stated  that 
"It  was  the  death  of  Lillenthal  that  brought  the  subject 
(the  problem  of  flight]  to  our  attention"  (io8:7).  This 
spurred  the  Wright  Brothers  to  write  to  the  Smithsonian 
Institution  in  order  to  obtain  books  relating  to  flight. 
Orville  Wright  contended  that  he  and  his  brother  Wilbur  had 
determined  "that  Lillenthal  had  been  killed  through  his 
inability  to  balance  his  machine  in  the  air"  (103:7). 
Further,  the  achievements  of  Samuel  Langley  and  Octave 
Chanute  also  advanced  the  theory  of  flight  (142:46).  In 
fact,  the  sheer  number  as  well  as  the  content  of  the 
letters  between  the  Wright  Brothers  and  Octave  Chanute 
indicates  that  Chanute  was  both  a  consultant  mentor  and 
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friend  of  the  Wright  Brothers  up  to  his  death  in  1910 
(108:7-537;  109:677-993). 

Clearly,  many  technical  risks  associated  with  powered 
heavier-than-air  flight  were  reduced  in  the  century  prior 
to  the  successful  Wright  Brothers  flight.  However, 
formidable  technical  risks  remained  for  the  Wright  Brothers 
to  solve.  The  Wright  Brothers  "took  up  gliding  as  a  hobby 
while  they  were  operating  a  bicycle  repair  shop  in  Dayton, 
Ohio"  (46:146).  A  perusal  of  the  content  of  the  papers 
(e.g.f  letters  and  diaries  of  the  Wright  Brothers)  reveals 
that  from  the  beginning  that  they  were  meticulous  in  their 
study  of  the  available  literature  relating  to  flight  and  in 
surveying  the  results  of  previous  flight  experiments 
(108:5, 19,20-22) . 

During  1900  and  1901  the  Wright  Brothers  tested 
gliders  in  the  steady  winds  of  Kitthawk,  North  Carolina. 

At  first  these  gliders  were  flown  like  box  kites,  later 
they  were  flown  manned — usually  pushed  from  a  small  hill. 
Wilbur  Wright's  notebooks  are  a  testament  to  the  methodical 
and  meticulous  nature  of  the  Wright  Brothers  investigations 
into  flight.  His  September  to  October  1900  notebook 
entries  contains  detailed  observations  that  he  made 
regarding  the  flight  of  birds.  In  particular,  the  flight 
characteristics  of  different  types  of  birds  were  compared. 
Wilbur  Wright  made  the  simple  observation  that  "no  birds 
soar  in  a  calm"  (108:7)  and  indicated  that  he  was  paying 
particular  attention  to  the  equilibrium  (i.e.,  balance)  of 
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birds  in  flight  (108:34-36).  In  a  letter  dated  23 
September  1900  to  his  father  Bishop  Milton  Wright,  Wilbur 
Wright  fundamentally  outlined  the  technical  risk  approach 
regarding  the  nearly  finished  first  Wright  glider: 

My  idea  is  to  experiment  and  practice  with  a  view  to 
solving  the  problem  of  equilibrium  .  .  .  once  a 
machine  is  under  proper  control  under  all  conditions, 
the  motor  problem  [to  obtain  true  flight]  will  be 
quickly  solved.  (108:26) 

That  some  letters  indicated  that  Wilbur  Wright  gave 
priority  to  equilibrium  in  flight  because  once  equilibrium 
was  attained  "a  failure  of  motor  will  simply  mean  a  slow 
descent  and  safe  landing  instead  of  a  disastrous  fall" 
(108:26).  Further  risk  reduction  was  indicated  by  Wilbur's 
intention  to  conduct  all  experiments  relatively  close  to 
the  ground  and  to  build  his  plane  "to  sustain  five  times  my 
weight  and  am  testing  every  piece"  (108:26).  Perhaps  the 
most  telling  indication  of  the  Wright  Brothers' 
understanding  of  risk  is  the  statement  by  Wilbur  in  that  23 
September  1900  letter: 

The  man  who  wishes  to  keep  at  a  problem  long  enough  to 
really  learn  anything  positively  must  not  take 
dangerous  risks.  Carelessness  and  overconfidence  are 
usually  more  dangerous  than  deliberately  accepted 
risks.  (108:26) 

Therefore,  just  in  case  of  any  potential  crash  landing 
Wilbur  Wright  concluded  that  landing  on  the  sand  near  Kitty 
Hawk  would  lessen  the  risk  of  injury  (108:26).  Because  the 
gliders  initially  exhibited  little  lift,  the  Wright 
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Brothers  were  convinced  that  the  curvature  of  the  wing 
surfaces  was  wrong.  To  address  this  technical  problem  they 
built  a  six-foot-long  wind  tunrel  to  study  various  curved 
wing  surfaces  in  moving  air.  ^hey  tested  hundreds  of 
glider  wings  in  the  wind  tunnel  and  made  "the  first 
reliable  tables  of  air  pressures  on  wings"  (46:146).  This 
effort  along  with  their  quantitative  records  and  analyses 
of  their  glides  contributed  immensely  to  their  airplane 
design  knowledge  (108:119-170).  In  1902,  based  on  the 
result  of  their  wind  tunnel  experiments,  the  Wright 
Brothers  built  a  third  glider  with  wings  of  32  feet  long 
and  approximately  5  feet  across.  This  glider  was 
successful  and  allowed  the  Wright  Brothers  to  make  over  a 
thousand  manned  glider  flights — some  for  longer  than  one 
minute.  At  the  time  there  were  only  crude  propellers  with 
efficiencies  that  were  below  50  percent,  and  no  developed 
theory  for  the  design  of  an  aerodynamic  propeller.  The 
Wright  Brothers  were  "the  first  to  recognize  that  the 
propeller  is  basically  a  rotating  wing,  made  up  of  airfoil 
sections"  (5:364).  This  understanding  allowed  them  to 
design  a  propeller  that  was  70  percent  efficient,  and 
greatly  contributed  to  the  success  of  their  first  flight 
(5:3-23,363,364;  46:146-147). 

The  development  of  an  efficient  propeller  was  not  the 
only  technological  risk  that  the  Wright  Brothers  had  to 
overcome.  Because  no  automobile  engine  existed  that  could 
meet  their  combined  power  and  weight  requirements  as  a 
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power  plant  for  their  aircraft,  they  found  it  necessary  to 
develop  their  own  using  an  automobile  engine  as  a  model  to 
aid  their  design.  The  Wright  Brothers'  mechanic  Charles 
Taylor  was  responsible  for  the  detail  design  of  the  engine 
(114:56).  In  February  1903,  during  the  first  test  of  their 
engine,  the  aluminum  crankcase  cracked.  Two  months  later, 
a  replacement  crank-case  finished  casting  and  in  May  1903 — 
roughly  7  months  before  their  historic  powered  flight — 
their  four  cylinder  (in-line)  engine  successfully  passed 
its  tests  (5:367;  43:237;  114:55). 

Rocket  Development.  In  July  1914,  Goddard  obtained 
"patents  on  rocket  combustion  chambers,  nozzles,  [and] 
propellant  feed  systems"  (5-372).  In  1915,  Robert  H. 
Goddard  demonstrated  the  principles  of  rocket  propulsion  in 
a  vacuum  at  Clark  university  in  Massachusetts.  In  November 
of  1923,  Goddard  achieved  successful  operation  of  a  rocket 
motor  in  a  test  stand  using  liquid  oxygen  and  gasoline 
jointly  as  the  rocket  fuel.  However,  there  was  clearly 
technical  risk  since  Goddard's  early  attempts  to  launch  a 
liquid— fueled  rocket  were  unsuccessful.  Further,  it  was 
necessary  for  Goddard  to  develop  specialized  high  pressure 
pumps  to  force  the  fuel  into  the  rocket  chamber.  Finally, 
on  16  March  1926,  Goddard  successfully  launched  the  world's 
first  liquid-fueled  rocket  engine.  This  success  in  the 
field  of  rocketry  was  analogous  to  the  Wright  Brothers' 
first  successful  flight  of  the  airplane  at  Kittyhawk,  North 
Carolina  in  December  1903  (5:1,3,372;  42:4,16,17,21; 
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145:48).  Robert  H.  Goddard's  development  of  the  rocket  by 
transforming  all  the  preceding  theoretical  rocket  knowledge 
into  actual  practice,  has  been  called  "one  of  the  most 
amazing  lone-wolf  development  programs  in  the  history  of 
technology"  (145:48). 

Almost  continuously  from  1930  to  1941,  Goddard 
conducted  development  and  flight  tests  at  Rosswell,  New 
Mexico  with  his  wife  and  four  assistants..  According  to 
Goddard's  wife,  the  "reliability  of  propulsion,  stability 
in  flight,  and  recovery,  (of  the  rocket  after  launch],  were 
the  primary  aims  in  the  early  tests,  rather  than  the 
attainment  of  altitude"  (145:46).  However,  to  further 
reduce  technical  risks  connected  with  rockets,  particularly 
as  it  regards  reliability,  extensive  mathematical 
reliability  modeling  had  to  be  employed. 

The  early  development  of  mathe  tical  reliability 
models  began  during  WWII  in  Germany,  where  a  group  led 
by  Werner  von  Braun  was  developino  the  VI  missile.  The 
first  series  of  ten  missiles  was  totally  unreliable; 
they  all  blew  up  on  the  launching  pads  or  fell  into 
the  English  Channel.  (89:1) 

This  underscores  the  point  that  ir  order  to  minimize 
technical  risks  of  new  technology  development  programs, 
special  analytical  or  mathematical  techniques  may  have  to 
be  refined  or  developed,  and  then  applied  (89:2).  A  brief 
chronology  of  the  development  of  the  rocket  is  provided  in 
Appendix  D.  A  detailed  chronology  of  Robert  H.  Goddard's 
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flight  tests  and  results  can  be  found  in  Von  Braun  and 
Ordway ' s  History  of  Rocketry  and  Space  Travel  (145:48-52). 

Atomic  Bomb.  The  critical  impact  of  experts  in  the 
reduction  of  technical  risk  is  readily  apparent  in 
connection  with  the  Manhattan  Project--the  United  States' 
project  that  resulted  in  the  world's  first  atomic  bomb.  In 
1939  Albert  Einstein,  at  the  urging  of  a  number  of 
scientists,  sent  a  letter  to  President  Franklin  D. 

Roosevelt  which  expressed  concern  that  Nazi  Germany  could 
be  making  significant  strides  towards  the  development  of 
the  first  atomic  weapon  (41:397).  President  Roosevelt 
established  the  Office  of  Scientific  Research  and 
Development,  and  appointed  the  United  States  scientist 
Vannevar  Bush  as  the  director.  In  December  1941,  the 
United  States  entered  World  War  II.  In  May  1942,  "The 
decision  [was]  made  to  proceed  on  all  promising  production 
methods"  (45:325)  for  obtaining  fissionable  materials. 
Vannevar  Bush  decided  to  involve  the  Army  in  the 
construction  of  the  plants  that  would  be  used  to  produce 
fissionable  materials.  Therefore  in  1942,  the  Army  Corps 
of  Engineer  opened  a  New  York  City  office  called  the 
Manhattan  Engineer  District  Office  under  General  Leslie 
Groves.  However,  on  1  December  1942,  when  General  Groves 
signe '*  the  contract  for  the  construction  of  a  facility  to 
produce  plutonium,  there  were  still  many  unknowns.  Most 
notably,  "the  diffusion  barrier  [used  to  separate  gases] 
had  not  yet  been  proven  practical  [and]  plutonium 
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chemistry  was  almost  unknown"  (45:325).  Although  a  nuclear 


fission  reaction  had  been  achieved  on  2  December  1942  in  a 
nuclear  reactor  at  the  University  of  Chicago,  formidable 
technical  obstacles  remained  ir>  the  path  of  atom  bomb 
development.  One  of  the  major  technical  challenges  was  to 
produce  enough  plutonium  and  to  separate  Uranium-235  (U- 
235)  from  Uranium-238  (U-238) .  A  laboratory  at  Los  Alamos 
(New  Mexico) ,  for  exploring  the  methods  of  manufacturing 
the  fissionable  materials  for  an  atomic  bomb  was  placed 
under  the  direction  of  J.  Robert  Oppenheimer  (46:517; 
47:831)  . 

Another  major  technical  concerns  regarded  the  approach 
that  should  be  pursued  in  the  atom  bomb  assembly.  During 
the  fall  and  summer  of  1943,  the  "gun"  approach  was  being 
pursued  for  the  design  of  a  nuclear  weapon.  Under  the  gun 
approach,  "a  mass  of  Uranium-235  or  (Plutonium  239)  would 
be  fired  into  .  .  .  another  piece  of  Uranium-235"  (45:325). 

When  the  two  pieces  joined  they  would  become  since 
"neutrons  would  be  created  at  a  faster  rate  than  they  can 
escape  from  the  assembly"  (45:324)  resulting  in  an 
explosive  release  of  energy  (45:324-325).  In  April  1943, 
Seth  Neddermeyer,  a  physicist  under  J.  Robert  Oppenheimer, 
"proposed  to  assemble  a  supercritical  mass  from  many 
different  directions"  (45:325)  instead  of  the  two 
directions  used  in  the  gun  approach.  Neddermeyer ' s 
proposal — the  implosion  method — was  supported  by  United 
States  physicist  Edward  Teller  and  United  States 


75 


mathematician  John  von  Neumann.  In  July  1944  it  was 
concluded  that  the  implosion  method  was  the  appropriate 
one.  However,  there  were  significant  technical  hurdles  to 
overcome  in  order  to  implement  the  implosion  method 
(45:325).  In  particular, 

The  big  problem  facing  the  physicists  at  Los  Alamos 
[New  Mexico]  was  how  to  produce  an  extremely  fast 
reaction  in  a  small  amount  of  Uranium  isotope,  U-235, 
or  of  plutonium  so  that  a  great  amount  of  energy  would 
be  explosively  released.  (81:180) 

John  von  Neumann,  a  theoretical  mathematician  who  also 
had  a  crucial  knowledge  of  hydrodynamics,  was  brought  in  as 
a  con— to  the  group  at  Los  Alamos  late  in  1943.  Von 
Neumann's  contributions  ultimately  resulted  in  the  success 
of  the  sophisticated  technique  of  attaining  critical  mass 
of  the  bomb's  nuclear  material  by  subjecting  such  material 
to  a  spherical  shock  wave.  Further,  von  Neumann  and  James 
L.  Tuck  invented  "an  ingenious  type  of  high  explosive  lens 
that  could  be  used  to  make  a  spherical  wave"  (81:181). 
However,  von  Neumann's  greatest  contribution  towards 
reducing  the  technical  risk  of  atom  bomb  development  was 
"his  showing  the  theoretical  people  how  to  model  their 
phenomena  mathematically  and  then  to  solve  the  resulting 
equations  numerically"  (81:181)  on  the  computer  (42:44; 

81: 178-181) . 

Indeed  technical  risks  associated  with  developing  the 
bomb  were  also  significantly  reduced  by  the  emergence  of 
the  first  electronic  computer--the  Electronic  Numerical 
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Integrator  and  Computer  (ENIAC) .  A  number  of  crucial 
computations  for  atom  bomb  development  were  done  with  the 
ENIAC,  and  other  computers,  with  the  aid  of  von  Neumann 
(81: 117 , 150)  . 

Another  project  in  which  von  Neumann  was  involved 
resulted  in  the  now  famous  Monte  Carlo  method. 
Fundamentally,  the  Monte  Carlo  method  involves  the  modeling 
of  a  system  as  a  repeated  probabilistic  experiment  in  order 
to  evaluate  the  collective  effect  of  probability 
distributions  of  the  individi  1  components  of  the  system. 
Von  Neumann  used  the  Monte  Carlo  Method  for  studying  the 
diffusion  and  multiplication  of  neutrons.  This  reduction 
of  the  complex  neutron  interaction  to  a  relatively  simple 
Monte  Carlo  simulation  gave  the  Los  Alamos  scientists  a 
valuable  tool  for  experimentation  (78:652-653;  81:294-296). 

The  fissionable  materials  were  manufactured  mainly  at 
Oak  Ridge,  Tennessee  and  at  Richland  near  Hanford, 
Washington,  and  then  shipped  to  the  Manhattan  Project 
research  laboratory  at  Los  Alamos,  New  Mexico  (46:517; 
47:831).  A  brief  chronology  of  events  regarding  the 
development  of  the  Atomic  Bomb  is  presented  in  Appendix  E. 

Laser  Development.  In  .917  Albert  Einstein  recognized 
the  principle  of  stimulated  emission.  The  principle  of 
microwave  amplification  by  stimulated  emission  of  radiation 
(MASER)  was  discovered  by  Charles  H.  Townes  of  Columbia 
University.  In  1954  a  working  maser  device  was  built  by 
Townes,  James  P.  Gordon  and  H.  J.  Zeiger,  and  subsequent 
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improved  models  were  built  by  Bell  Laboratories.  Since  the 
masers  operated  at  microwave  frequencies,  Arthur  L. 

Schawlow  and  Charles  H.  Townes  proposed  a  technique  to 
obtain  a  maser-like  device  which  operated  at  optical 
frequencies.  Assuming  that  the  technical  risk  could  be 
overcome,  the  device  in  question  would  in  actuality  be 
based  on  light  amplification  by  stimulated  emission  of 
radiation  (LASER).  The  first  successful  construction  and 
operation  of  a  laser  was  achieved  in  1960  by  T.  H.  Maiman 
of  the  Hughes  Aircraft  Company  using  a  ruby  crystal 
(44:686;  134:234;  135:227). 

Schawlow  states  that  the  building  of  an  optical  maser 
(i.e.,  a  laser)  "required  preparation  of  an  active  medium 
that  would  actually  display  maser  action  in  the  optical 
region  of  the  spectrum"  (135:228).  One  of  the  major 
problems  in  using  crystals  or  glasses  as  the  lasing  medium 
is  that  they  "are  usually  formed  at  high  temperatures  and 
require  considerable  effort  and  expense"  (101:255)  to  free 
them  of  optical  imperfections.  The  ruby  crystal  in 
Maiman 's  laser  had  been  machined  into  a  rod  whose  ends  were 
"polished  optically  flat  and  parallel  and  .  .  .  partially 

silvered"  (135:228).  To  obtain  lasing,  the  ruby  rod  was 
then  subjected  to  light  from  an  electronic  flash  tube  which 
could  attain  the  necessary  critical  intensity  (44:686).  A 
brief  chronology  of  laser  development  is  presented  in 
Appendix  F. 
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Project  Apollo.  On  25  May  1961,  President  John  F. 
Kennedy  declared  that  the  United  States  "should  commit 
itself  to  achieving  the  goal,  before  this  docade  is  out,  of 
landing  a  man  on  the  moon  and  returning  him  safely  to 
earth"  (116:736).  A  prerequisite  for  achieving  the  goal 
was  the  development  of  a  booster--the  Saturn  V — capable  of 
the  required  thrust  (91:38). 

Saturn  v  Rocket.  The  three-stage  Saturn  V  rocket 
was  the  product  of  highly  experienced  rocket  experts  who 
"had  lost  a  number  of  rockets  in  earlier  development 
programs  and  had  blown  up  a  few  engines"  (90:52).  The 
rocket  expert  Werner  von  Braun  had  accumulated  data  from 
launching  V-2  missiles  in  the  New  Mexico  desert  and  then 
began  building  improved  successors  to  the  V-2.  Other  engine 
experts  such  as  David  E.  Aldrich  of  Rocketkdyne,  who  "had 
helped  to  develop  the  engine  for  the  X-2  research  aircraft" 
(90:52),  provided  necessary  expertise  for  the  Saturn  V 
development.  According  to  Aldrich,  since  "the  heart  of  [a 
jetj  engine"  (90:53)  the  fuel  injector,  that  is  the 
first  obstacle  tc  successful  engine  development.  Aldrich 
was  the  RocKetdyne  program  manager  for  the  Saturn  V  F-l 
engine.  An  early  technical  problem  with  the  F-l  engine  was 
that  the  fuel  combustion  "was  so  violent  that  it  triggered 
shock  waves  producing  more  heat  than  the  engine's  cooling 
system  could  handle"  (90:53).  On  28  June  1962  one  F-l 
engine  was  destroyed  when  to  the  "fire  in  the  (combustion] 
chamber  burned  through  the  injector"  (90:53).  Within  a 
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span  of  one  and  a  half  years,  thirty  different  injector 
designs  were  tried  without  success.  Finally,  the  test 
engineers  decided  to  use  explosive  charges  within  the 
coirbustion  chamber  to  trigger  shock  waves  when  the  rocket 
was  firing.  It  was  thereby  discovered  that  when  baffles 
were  used  to  isolate  each  combustion  region,  the  shock 
waves  in  the  chamber  would  die  out.  Other  refinements  such 
as  enlarging  the  diameters  of  the  holes  from  which  the  fuel 
and  oxidizer  were  squirted,  finally  resulted  in  a  solution 
(90:52,53) . 

The  size  and  weignt  of  the  Saturn  V  was  a  major 
challenge  since  it  stands  "363  feet  tall  [and  when]  fully 
fueled  it  weighs  nearly  6.4  million  tons"  (145:171).  New 
building  techniques  had  to  be  used  for  the  Saturn  V,  and 
facilities  capable  of  withstanding  the  weight  of  the 
tooling  were  necessary.  One  of  the  major  trade-off 
decisions  that  had  to  be  made  for  Project  Apollo  was  thut 
of  structure  versus  weight.  The  need  to  reduce  the  weight 
became  so  critical  that  the  second  stage  of  the  Saturn  V 
"emerged  as  the  merest  eggshell  of  a  rocket"  (90:99).  This 
fact  contributed  to  the  rupture  of  two  Saturn  second  stages 
during  test  (90:99;  145:170-173). 

The  first  launch  of  a  Saturn  V  on  9  November  1966  was 
successful.  Due  to  a  decision  by  George  E.  Mueller, 
director  of  NASA's  Office  of  Manned  Space  Flight,  live 
upper  stages  for  the  Saturn  V  were  used  on  these  unmanned 
(i.e.,  test)  launches.  On  the  second  launch  in  April  1968, 
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Two  of  the  J-2  engines  in  the  second  stage  shut  down 
prematurely.  The  J-2  engine  in  the  third  stage  failed 
to  re-start  on  command,  leaving  the  stage  stranded  in 
orbit.  (90:60) 

These  failures  during  the  second  Saturn  V  launch  were 
of  major  concern  because  the  next  launch  had  been  scheduled 
to  be  manned.  The  malfunction  investigation  of  the  second 
flight  was  confounded  by  the  fact  that  the  failed  hardware 
was  inaccessible — either  in  orbit  or  burned  up  upon  re¬ 
entry.  The  telemetry  (i.e.,  transmitted  data)  that  had 
been  obtained  from  the  prior  to  the  malfunctions  was  scanty 
since  there  were  a  relatively  small  number  of  Saturn  V  data 
channels.  Further,  these  particular  in  flight  failures  had 
never  been  encountered  during  ground  tests.  The  program 
manager  Paul  Castenholtz  and  Marshall  McClure  had  been 
trying  to  determine  why  the  failures  occurred  in  space  but 
not  on  the  ground.  By  repeatedly  studying  movies  of  J-2 
engines  firing  on  the  test  stands  they  began  to  realize 
that  the  ice  that  formed  on  the  fuel  and  oxygen  lines 
during  ground  tests  was  highly  significant.  The  hypothesis 
was  that  the  ice  actually  dampened  some  severe  vibrations 
and  thereby  prevented  the  fuel  and  oxygen  lines  frot,i 
rupturing.  Using  the  available  telemetry  data  Castenholtz 
localized  the  problem  to  a  special  part  of  the  engine. 

Then,  eight  of  the  suspect  fuel  lines  were  operated  in  a 
special  test  chamber  which  duplicated  the  firing  of  a  J-2 
engine  in  the  vacuum  of  space.  All  eight  fuel  lines 
ruptured.  Since  the  ice  that  formed  on  the  fuel  lines 
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during  ground  tests  did  not  form  in  space,  the  vibrations 
were  not  dampened  in  space.  This  was  the  reason  that  a 
fuel  line  could  break  "after  a  few  minutes  of  vibration  in 
space"  (90:60).  The  problem  was  solved  by  substituting  a 
"rigid  stainless  steel  pipe"  (90:60)  for  the  flexible  fuel 
lines.  "Thousands  of  man-hours"  (145:171)  were  required  to 
identify  and  correct  the  stated  malfunction.  The  cause  of 
both  J— 2  engine  failures  during  the  second  Saturn  V  launch 
was  specifically  attributed  to  the  "fatigue  failure  of  a 
small  liquid  hydrogen  line"  (145:240). 

To  reduce  the  enormous  technical  risks  associated  with 
landing  a  man  on  the  moon  and  returning  him  safely  to 
earth,  the  United  States  implemented  a  series  of 
spaceflights  designated  as  Project  Apollo.  One  of  the 
major  technical  decisions  involved  the  actual  flight  path 
and  moon  landing  technique  that  would  be  followed  in  order 
to  land  on  the  moon  and  return.  It  was  known  that  the 
flight  profile,  in  turn,  would  govern  the  "configuration  of 
the  entire  vehicle"  (11:33).  There  were  strong  arguments 
between  "NASA  and  the  academic  science  community"  (11:33) 
regarding  this  issue  of  the  flight  path  and  the  moon 
landing  technique.  In  late  1962  a  technique  called  the 
Lunar  Orbital  Rendezvous  (LOR)  method  was  generally 
accepted.  Under  the  LOR  method,  the  spacecraft  would  orbit 
the  moon  upon  arrival.  Then,  one  section  would  detach  from 
the  orbiting  spacecraft  and  land  on  the  moon's  surface. 

For  the  return,  the  detached  section  would  blast  off  from 
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the  moon's  surface  and  rendezvous  (join)  with  the  orbiting 
portion  of  the  spacecraft  (11:33,35). 

The  Apollo  spacecraft  consisted  essentially  of  three 
parts.  The  Command  Module  (CM)  contained  crew  controls  and 
living  space.  The  Service  Module  (SM)  contained  the 
propulsion  system,  fuel  and  other.  The  Service  Module  was 
connected  to  the  Command  Module  but  the  SM  was  jettisoned 
prior  to  re-entering  the  earth.  Finally,  the  Lunar  Module 
(LM)  had  an  ascent  and  descent  stage  that  was  used  to  carry 
the  astronauts  between  the  CM,  wh. ch  remained  in  lunar 
orbit,  and  the  lunar  surface.  All  of  this  sat  atop  the 
Saturn  V  rocket  during  launch  (91:40). 

Due  to  heavy  schedule  pressure,  NASA  management 
decided  to  do  "all-up"  flight  testing.  Instead  of  testing 
component  by  component,  gradually  building  confidence  in 
the  system,  the  full  apparatus  would  be  tested  in  ready-to- 
fly  configuration.  This  all  up  approach  was  considered  to 
be  unorthodox  engineering  (11:35). 

Gemini  Space  Vehicles.  The  Apollo  program  had 
been  preceded  by  the  considerable,  but  less  ambitious. 
United  states  space  endeavors  such  as  Project  Mercury  which 
was  the  first  United  States  manned  space  program,  and 
Project  Gemini  (91:80-82,65-67).  Project  Gemini  was  the 
successor  to  Project  Mercury.  The  Gemini  series  of  space 
vehicles  (12  in  the  series)  provided  the  astronauts  and 
ground  crew  valuable  experience  in  activity  (i.e., 
spacewalks) ,  rendezvous  and  docking,  long  duration  flights, 
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and  guided  re-entry.  This  experience  was  to  prove 
invaluable  (91:65).  In  fact,  the  first  in-space  emergency 
occurred  during  Gemini  8.  On  16  March  1966  the  Gemini  8 
spacecraft  with  Armstrong  and  Scott  achieved  the  first  in¬ 
space  docking  with  an  Agena  target  rocket  placed  in  orbit 
101  minutes  earlier"  (91:66).  After  linking,  the  two 
vehicles  started  "tumbling  and  spinning  out  of  control" 
(91:66)  due  to  a  jammed  Gemini  thruster.  Armstrong  and 
Scott  "escaped  only  by  firing  their  retro-rockets"  (91:65) 
and  returned  to  earth  two  days  earlier  than  planned.  Also, 
tragically,  t.he  original  crew  of  the  Gemini  9,  See  and 
Basset,  died  while  trying  to  land  "a  jet  fighter  in  bad 
weather  at  McDonnell  works"  in  St.  Louis  (91:67). 

The  Apollo  program  started  off  with  tragedy 

.  .  .  when  on  27  Jan  1967  a  short  circuit  in  the 

electrical  systems  set  fire  to  the  all-oxygen 
atmosphere  while  the  crew,  Virgil  Grissom,  Edward 
White  and  Roger  Chaffee,  were  carrying  out  a  full 
launch  rehearsal  on  Pad  34.  White,  reaching  back  over 
his  head  was  unable  to  open  the  hatch  in  the  few 
seconds  before  the  crew  was  overwhelmed.  (92:53) 

In  the  wake  of  the  tragedy,  major  re-design  of  the 
space-craft  cabin  and  marked  improvements  to  the  crew 
operations  were  undertaken  to  greatly  minimize  crew  safety 
risks  (77:71,271;  92:53). 

Site  Selection.  To  reduce  the  enormous  technical  and 
operational  risks  involved  in  achieving  a  roundtrip  manned 
moon  landing,  each  flight  in  the  Apollo  series  tackled 
increasingly  difficult  technical  and  operational 
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milestones.  Some  highlights  of  selected  Apollo  flights  are 
presented  below.  Technical  teams,  especially  geologists, 
devoted  5  years  looking  for  a  suitable  lunar  landing  site. 
Several  of  the  flights  in  the  Apollo  series,  particularly 
those  which  orbited  the  moon,  had  an  objective  of  observing 
possible  lunar  landing  sites.  Such  detailed  surveillance 
and  selection  of  potential  lunar  landing  sites  was 
necessary  to  curtail  the  risk  that  the  moon  lander  would 
slide  off  a  cliff,  or  land  on  uneven  ground  and  overturn. 

It  was  further  necessary  to  verify  that  the  surface  was 
firm  enough  so  that  the  lander  would  not  sink  and  be 
swallowed  up  upon  landing.  Some  of  the  stated  technical 
risks  connected  with  the  lunar  surface  were  addressed  by 
the  series  of  Ranger  and  Surveyor  spacecrafts.  Rangers  1 
and  2  were  orbited  around  the  Earth  "to  check  out 
instrumentation"  (145:191).  Rangers  3  through  6,  intended 
to  explore  the  moon,  failed,  but  in  1964  and  1965,  Rangers 
7,  8  and  9  relayed  some  17,000  images"  (150:44)  of  the 
moon's  surface  before  crash  landing  onto  the  moon.  The 
pictures,  with  resolution  of  1  foot,  showed  rocks  on  the 
moon's  surface  and  indicated  that  the  surface  could  support 
some  heavy  objects  (145:191;  150:43,44). 

However,  site  selection,  had  to  take  into  account 
other  risk  factors  such  as  "communication,  tracking,  fuel 
capacity,  rocket  performance"  (150:44).  When  the  Soviet 
Union's  Luna  9  and  the  United  States'  Surveyor  I  probes 
both  landed  on  the  moon  in  1966,  the  moon's  surface  was 
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"barely  dented"  (150:45) .  Follow-on  Surveyor  probes 
established  the  properties  and  chemical  composition  of  the 
lunar  surface.  Once  it  was  determined  in  what  areas  to 
concentrate  the  search  for  landing  sites,  a  series  oi  five 
Boeing-built  Lunar  Orbiters  were  sent  to  photograph  the 
lunar  surface  so  that  by  February  1968,  five  potential 
lunar  landing  sites  were  selected  by  relying  on  the 
expertise  of  geologists.  The  top  two  candidate  sites  were 
then  observed  by  astronauts  in  Apollo  8  and  10  during  lunar 
orbits  as  low  as  10  miles  above  the  lunar  surface.  The 
information  obtained  from  the  Surveyor  and  Orbiter  probes 
was  crucial  for  the  planning  of  the  Apollo  manned  mission 
(145: 193;  150:46,47) . 

Simulator  Training.  While  the  geologists  were 
searching  for  a  safe  and  suitable  landing  site,  the 
astronauts  were  training  for  the  actual  moon  landing 
operations.  In  order  that  the  astronauts  could  simulate 
landing  on  the  moon,  a  Lunar  Landing  Training  Vehicle 
( LLTV )  was  built.  The  LLTV  was  a  truss  assembly  (no  wings) 
built  around  a  vertically  placed  jet  engine.  The  thrust 
allowed  vertical  takeoff  and  landing.  When  the  LLTV  thrust 
was  adjusted  to  support  five-sixth  of  the  vehicle  weight, 
the  LLTV  could  be  used  to  effectively  simulate  the  effect 
of  landing  in  lunar  gravity  from  about  500  feet  above  the 
moon's  surface.  "To  prepare  the  astronauts  for 
emergencies,  the  LLTV  was  occasionally  programmed  to  fail" 
(150:44).  There  were  a  number  of  occasions  when  an  LLTV 
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pilot  had  to  eject  from  a  malfunctioning  LLTV .  When  the 
attitude  control  rockets — which  kept  the  flight  stable — 
shut  down  putting  the  LLTV  into  a  spin,  Neil  Armstrong 
ejected  at  200  feet.  Another  NASA  pilot  had  to  eject 
during  another  attitude  control  failure,  while  yet  another 
pilot  ejected  from  an  LLTV  that  was  blown  out  of  control  by 
strong  winds  (150:44). 

Apollo  7.  The  October  1968  Apollo  7  flight, 
although  confined  to  Earth  orbit,  was  the  first  manned 
flight  of  the  Command  and  Service  vehicles.  During  Apollo 
7,  the  main  propulsion  engines  of  the  Service  module  were 
fired  automatically  as  well  as  manually,  and  the  engine 
performance  was  evaluated.  Further,  the  heat  shield  of  the 
vehicle  was  evaluated  during  re-entry  (92:53). 

Apollo  8.  While  the  Apollo  7  operations  were 
confined  to  Earth  orbit,  the  December  1968  Apollo  8  flight 
was  man's  first  flight  around  the  moon.  In  fact,  it  was 
the  first  time  that  the  3000  ton  Saturn  5  rocket  had  been 
used  to  launch  men  into  space.  During  the  147-ncur  Apollo 
8  mission,  the  crew  achieved  the  critical  Lunar  orbit 
insertion.  They  spent  20  hours  circling  the  moon,  and 
photographed  potential  landing  areas  (92:53;  149:1). 

Apollo  9.  Apollo  9  continued  the  trend  of 
incrementally  reducing  the  technical  risks  associated  with 
landing  a  man  on  the  moon.  The  March  1969  Apollo  9  flight 
was  confined  to  Earth  orbit,  however,  it  was  the  first 
manned  flight  with  the  Luuar  Module.  During  the  ten  day 
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flight,  separation,  rendezvous  and  docking  was  practiced 
between  the  Lunar  Module  (LM)  and  the  Command  Module  (CM) . 
Although  the  flight  was  confined  to  earth  orbit,  some 
operations  were  nevertheless  performed  to  simulate  a  moon 
lift-off  (92:53).  Apollo  9  the  May  1969  Apollo  10  flight 
was  "a  successful  dress  rehearsal  for  the  Moon  landing 
[that  occurred]  two  months  later"  (92:53).  During  the 
Apollo  10  flight,  the  entire  Apollo  spacecraft  orbited  the 
moon  for  the  first  time.  To  simulate  a  moon  landing,  the 
Command  Module  maintained  a  111  kilometer  orbit  around  the 
moon,  while  the  Lunar  Module,  with  Stafford  and  Cernan  on 
board,  separated  and  descended  twice  to  nearly  14.5 
kilometers  from  the  surface  of  the  moon.  The  Lunar  Module 
then  re-docked  with  the  Command  Module  after  a  separation 
of  8  hours  (92:53) . 

Apollo  11.  Finally,  the  July  1969  Apollo  11 
flight,  with  the  crew  of  Neil  Armstrong,  Edwin  Aldrin  and 
Mike  Collins,  resulted  in  the  first  men  r n  the  moon 
(148:1).  However,  despite  the  previous  Apollo  flights, 
there  remained  significant  sources  of  risk  that  were 
evident  during  the  Apollo  11  flight.  For  instance,  on  the 
way  down  to  the  moon  in  the  Lunar  Module  "Armstrong  decided 
to  take  over  manual  control  because  the  spacecraft  was 
approaching  an  area  in  the  Sea  of  Tranquility  strewn  with 
boulders"  (92:54).  In  fact,  "had  Armstrong  not  taken 
control  (the  craft]  might  have  overturned  or  smashed  on 
alighting"  (116:765).  Other  technical  risk  was  evident 
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since  it  was  confirmed  that  one  of  the  on-board  computers 
was  over-loading.  That  computer  "was  a  vital  part  of  the 
radar  system  which  calculated  the  LM's  [Lunar  Module's] 
attitude  and  rate  of  descent,  and  on  which  the  life  of  the 
crew  depended"  (92:54). 

Although  the  Apollo  11  flight  put  the  first  men  on  the 
moon  on  21  July  1969,  there  were  nevertheless  near 
disasters  on  some  of  the  later  moon  landing  flights.  This 
clearly  illustrated  that  the  sources  of  technical  risk,  were 
not  eliminated.  For  instance,  the  November  1969  Apollo  12 
mission 

.  .  .  started  sensationally,  for  as  the  launch  was 

being  made  through  a  rain  squall,  the  Saturn  5  rocket 
was  struck  by  lightning.  The  spacecraft's  electrical 
system  was  briefly  put  out  of  action  and  for  the  first 
time  during  a  manned  launch  they  were  very  close  tc  an 
abort.  (92:55) 

But,  the  flight  was  not  aborted  and  the  crew  members 
of  Apollo  12  went  on  to  complete  a  mission  that  resulted  in 
31.5  hours  of  walking  on  the  moon  (92:55).  Appendix  G 
provides  a  brief  chronology  of  the  achievement  of  a  man  on 
the  moon. 

Apollo  13  Accident.  The  April  1970  Apollo  13  mission 
had  been  planned  to  be  the  third  moon  landing  mission. 
However, 

.  .  .  an  explosion  on  board  when  the  spacecraft  was 

329,915  km  from  the  earth  all  but  cost  the  lives  of 
the  crew  and  turned  the  mission  into  a  3.5  day  rescue 
drama  ...  as  tens  of  thousands  of  technicians  worked 
to  bring  the  crippled  spacecraft  safely  home.  (92:56) 


Initially,  it  was  determined  that  due  to  the  explosion 
only  half  the  power,  water,  and  oxygen  that  was  needed  to 
get  the  crew  home,  would  be  available.  However,  based  on 
simulations  that  were  done  by  technicians  on  the  ground, 
some  ways  were  found  to  conserve  energy  by  powering  down 
systems.  This  unfortunately  meant  that  the  cabin  became 
increasingly  cold  and  disaster  threatened  again  .  .  when 

the  astronauts,  tired  and  chilled,  made  mistakes”  (92:56). 
For  the  journey  home,  it  was  necessary  for  the  astronauts 
to  move  into  the  Lunar  Module  and  use  it  like  their 
"lifeboat  (a  backup  craft  for  their  safe  return)  .  .  . 

using  a  jury-rigged  arrangement  ...  to  purge  carbon 
dioxide  from  their  atmosphere"  (92:56).  It  was  later 
determined  that  an  oxygen  tank  had  overheated  and  exploded 
since  some  "heater  switches  had  welded  shut  when  subjected 
to  excessive  pre-launch  electric  currents”  (92:57).  The 
exploding  tank  took  another  oxygen  tank  out  of  commission 
as  well.  Astronaut  Lovell — one  of  the  Apollo  13  crew 
members — concluded  that 

.  .  .  warning  signs  during  testing  went  unheeded,  and 

the  tank,  damaged  from  8  hours  of  overheating,  was  a 
potential  bomb  the  next  time  it  was  filled  with 
oxygen.  That  bomb  exploded  on  13  April  1970 — 200,000 
miles  from  Earth.  (92:57) 

This  illustrates  that  testing — intended  to  decrease 
technical  risk--if  done  incorrectly,  can  actually  increase 
technical  risk  by  causing  damage  (92:56,57;  147:1). 
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Space  Shuttle  Challenger  Disaster.  Inadequately 
managed  technical  risk  has  resulted  in  disasters  and 
national  defense  setbacks.  The  in-flight  explosion  of  the 
Space  Shuttle  Challenger  (Flight  51-L)  on  28  January  1986, 
resulted  in  the  death  of  seven  astronauts  on  board  the 
shuttle,  and  the  concomitant  loss  of  part  of  the  military 
space  operations  capability.  Investigations  revealed  that 
the  hot  exhaust  gases  in  a  solid  rocket  motor  of  the  space 
shuttle  eroded,  and  then  bypassed  two  "O-ring"  seals. 

These  hot  gases  eventually  ignited  the  hydrogen  in  the 
nearby  space  shuttle  External  Tank,  thereby  causing  an 
explosion.  However,  the  potential  for  just  such  a 
disastrous  O-ring  sealing  failure  on  the  solid  rocket 
motors  had  been  identified  as  early  as  17  December  1982  and 
the  failure  potential  of  the  seals  was  under  study  at  the 
time  of  the  disaster.  Indeed,  Morton  Thiokol,  Inc., 
producer  of  the  solid  rocket  motors,  had  recognized  the 
need  for  improvement  of  the  field  joints  as  well  as  the 
motor  case/nozzle  joint  prior  to  the  accident  (68: 3; 
102:38).  In  the  wake  of  the  Challenger  disaster,  nearly 
220,  155,  31  and  8  modifications  were  made,  respectively, 
to  the  Orbiter,  solid  rocket  motors,  main  engines  and 
external  tank.  In  particular,  a  third  "0-rir.g"  and  a  metal 
seal  feature  was  added  to  the  solid  rocket  motor  field 
joint  (12:1;  67:10,11;  93:24;  102:38,39;  103:24-26). 

Some  of  the  major  changes  that  were  made  to  the  Space 
Shuttle  solid  rocket  motors  to  reduce  technical  risk  were 
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actually  quite  mundane,  yet  very  important.  These  included 
the  following: 

Heaters  to  maintain  seal  temperature  at  75  degrees 
Fahrenheit  or  above,  a  weather  seal  to  prevent  water 
entry  into  the  joint  and  possible  freeze-up,  longer 
pins  [used  to  hold  the  rocket  segments  together  at 
each  joint]  and  new  retention  bands,  an  alternative 
insulation  (J-seal)  to  eliminate  the  putty  previously 
used.  (67:11) 

In  addition  to  the  hardware  changes  that  were  made,  a 
number  of  procedural  changes  were  recommended  by  the 
Presidential  commission  that  investigated  the  accident. 

The  commission  recommended  that  detailed  testing  of  the 
solid  rocket  motor  joint  be  conducted  in  the  "exact  flight 
configuration  in  the  vertical  position"  (67:10),  and  that 
the  National  Research  Council  oversee  the  NASA  re-design  of 
the  solid  rocket  booster  or  joints  (67:10-13). 

Other  commission  recommendations  included  an  increased 
emphasis  on  rotating  some  former  astronauts  into  management 
positions.  Further,  the  commission  recommended  that  a 
major  effort  be  undertaken  to  provide  a  crew  escape 
capability  and  an  improved  in-flight  launch  abort 
capability  for  the  space  shuttle.  Most  of  the  recommended 
changes  were  implemented  (67:32).  Thirty-two  months  after 
the  Challenger  disaster,  the  shuttle  named  Discovery,  with 
five  astronauts  aboard,  was  flown  to  a  successful  mission 
(93:20) . 

After  the  Challenger  disaster,  NASA  intensified  their 
three-step  risk  analysis  effort.  The  NASA  three-step  risk 
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analysis  effort  consists  of  a  Failure  Modes  and  Effects 
Analysis  ( FMEA) ,  a  Critical  Items  List  and  a  Hazard 
Analysis.  The  FMEA  is  conducted  to  identify  hardware  that 
are  "critical  to  performance  and  safety  of  the  vehicle  and 
mission"  (67:8).  As  the  second  step,  NASA  makes  a  Critical 
Items  List  (CIL)  which  identifies  those  components  whose 
failure  alone  (i.e.,  a  single  failure)  would  cause  either 
loss  of  life,  vehicle  or  mission.  The  CIL  contains  the 
rationale  for  those  specific  components  in  tha  design  that 
do  not  have  redundancy.  Such  rationale  should  provide 
"information  regarding  the  (1)  design,  (2)  tests 
accomplished  to  assure  integrity  of  the  hardware,  (3) 
specific  inspection  points,  and  (4)  operational  means  to 
mitigate  the  failure"  (67:8).  The  CIL  is  based  on  the 
results  of  the  FMEA.  As  the  third  step,  NASA  conducts  a 
Hazard  Analysis  (HA) .  The  purpose  of  the  HA  is  to  identify 
and  recommend  necessary  corrective  actions  for  those 
potential  sources  of  risk  that  arise  during  the  operation 
and  maintenance  of  the  system  hardware  and  software.  The 
HA  also  examines  sources  of  risk  arising  from  man/machine 
interfaces  (e.g.,  the  crew),  the  environment,  and 
mission— related  activities  (67:7-9). 

Inertial  Upper  Stage  Anomaly.  Three  years  before  the 
Challenger  disaster,  on  5  April  1983,  hot  exhaust  gases 
were  also  cited  for  eroding  and  thereby  bypassing  seals  on 
the  Solid  Rocket  Motor  #2  (SRM-2)  of  the  Inertial  Upper 
Stage  (XUS)  booster.  This  ultimately  resulted  in  a  mission 
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anomaly  whereby  a  NASA  satellite — the  Tracking  Data  Relay 
Satellite  (TDRS)--was  placed  in  an  incorrect  orbit.  The 
combined  TDRS  satellite  and  the  IUS  booster  had  been 
released  from  the  payload  bay  of  the  Space  Shuttle  in  low 
Earth  orbit  (less  than  200  miles  above  Earth).  During  this 
sixth  space  shuttle  mission  the  IUS  was  supposed  to  boost 
the  satellite  to  the  much  higher  geosynchronous  orbit. 
However,  the  combined  satellite  and  IUS  booster  began  to 
tumble  in  space  about  83  seconds  after  the  start  of  the 
SRM-2  burn.  Specifically  the  Inertial  Upper  Stage  Anomaly 
Board  investigation  concluded  that  the  failure  resulted  in 
an  IUS  ''nozzle  gimble  mechanical  failure"  (4:4)  that 
resulted  in  the  SRM-2  nozzle  being  jammed  in  an  off-center 
position.  Due  to  the  mechanical  failure  the  nozzle 
positioning  actuators  were  unable  to  correctly  reposition 
the  nozzle  until  the  SRM-2  motor  burned  out  (i.e.,  finished 
thrusting) .  The  author  of  this  thesis  was  in  the  Inertial 
Upper  Stage  Systems  Program  Office  at  the  time  (4:4). 
However,  the  engineers  were  later  able  to  successfully 
execute  a  plan  to  command  the  satellite  to  fire  its  small 
rocket  thrusters  and  finally  achieve  the  correct  orbit 
(4:4;  120:116) . 

In  fact,  the  technical  risk  from  "flame  erosion" 
(66:39)  caused  by  the  leakage  of  hot  gases  has  even  been 
manifest  after  post-firing  inspections  of  the  Phoenix 
missile  in  April,  1986  (66:39). 
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AFOS  Software  Development.  A  1982  GAO  report  cited 
that  the  National  Weather  Service's  automated  data 
processing  and  telecommunications  system  was  undergoing 
some  major  technical,  scheduling  and  managerial  problems 
relating  primarily  to  software  development.  The  system  is 
called  the  Automation  of  Field  Operations  and  Services 
(AFOS) .  One  of  the  problems  cited  was  that  the  "AFOS 
hardware  lacks  sufficient  core  memory  to  accommodate 
current  software  or  applications  initially  planned  for 
AFOS;  the  Weather  Service  cannot  tell  ho w  much  more  memory 
is  neede"  (65:ii).  In  addition,  the  operating  system 
could  not  "meet  concurrent  processing  requirements 
originally  specified"  (65:ii). 

The  GAO  cited  that  one  of  the  major  problems  was  that 
the  software  was  "unnecessarily  complex"  (65:31).  For 
example,  each  of  the  software  modules  in  the  AFOS  were  made 
so  interdependent  that  even  minor  changes  to  one  module 
caused  major  technical  repercussions  to  the  other  modules. 
This  module  interdependence  also  complicated  the 
troubleshooting  of  programs.  In  addition,  the  AFOS  system 
would  experience  "deadlock  when  the  computer  [attempted]  to 
process  two  or  more  tasks  that  need  the  same  resources" 
(65:31).  Also,  information  that  was  being  input  would  be 
lost  if  the  system  malfunctioned  during  the  inputting 
(65:31,32).  One  of  the  primary  causes  of  problems  with  the 
National  Weather  Service  (NWS)  software  was  NWS 
inexperience  with  software  projects  of  that  magnitude.  Due 
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to  inexperience,  the  NSW  had  erroneously  assigned  a  higher 
priority  to  the  data  processing  hardware  than  to  the 
software  development  procedures  (65:31).  In  fact,  the  GAO 
found  that  "development  priority  was  not  clearly 
established  and  top  priorities  (were)  ignored  in  favor  of 
low  priority  work"  (65:21). 

Changing  user  requirements  meant  that  completed 
software  work  had  to  be  redone.  A  large  amount  of 
technical  risk  also  stemmed  from  inadequate  design 
specifications.  With  only  ambiguous  specifications  to 
guide  them,  each  programmmer  tended  to  act  independently 
rather  than  as  part  of  a  coordinated  team.  An  outgrowth  of 
this  was  that  the  software  documentation  so  "critical  to 
effective  software  development  and  operation"  (65:35)  was 
clearly  deficient  (65:31-35). 

Space  Defense  Initiative  (SDI) 

The  Space  Defense  Initiative  (SDI)  is  a  "program  to 
determine  the  feasibility  of  developing  and  deploying  a 
defense  against  nuclear  ballistic  missiles"  (72:1).  To 
reduce  the  inherent  technical  risks,  the  SDI  organization 

has  used  ...  3  parallel  contractor  efforts  to 
analyze  technology,  propose  preliminary  system 
concepts  and  associated  mission  performance  and 
interface  requirements,  define  data  needs  for  sensor 
design,  and  develop  preliminary  demonstration  plans. 
(72:4) 

The  technical  risk  reduction  strategy  include;,  the 
conduct  of  some  "technology  validation  experiments"  (71:1) 
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by  the  Army's  Strategic  Defense  Command.  The  objective  is 
to  demonstrate  that  issues  regarding  critical  technologies 
are  resolved  so  that  the  risks  in  any  ensuing  full  scale 
development  phase  are  also  reduced.  For  instance,  to 
ensure  that  effective  sensors  for  a  strategic  defense 
system  can  in  fact  be  developed,  the  Army  had  initially 
pursued  two  competing  sensor  designs.  In  fact,  it  has  been 
determined  that  three  types  of  sensors---corresponding  to 
the  three  trajectory  phases  of  launch,  midcourse  and 
terminal--would  be  necessary.  Technical  risk  has  also  been 
identified  in  connection  with  the  design  of  a  sophisticated 
data  processor  and  with  the  air  turbulence  expected  during 
airborne  tests  of  sensors  (6:94;  71:12).  Key  technologies 
have  been  identified  for  SDI  such  as 

1.  component  technologies  for  long-wave  infrared 
sensors  including  optics,  detectors,  signal 
processors,  and  cryogenic  coolers,  technologies 
for  laser 

2 .  technologies  for  laser  radars  and  space-based 
radars.  (72:3) 

However,  it  is  clear  that  a  host  of  experimental  or  new 
technologies  may  eventually  be  necessary  to  achieve  an 
actual  strategic  defense.  As  a  result,  the  Livermore 
National  Laboratory  is  carrying  out  X-ray  laser  research  in 
support  of  SDI  (69:1,2).  It  is  possible  that  space  nuclear 
rocket  technology  will  become  very  important  for  SDI 
applications  (64:8,11).  The  SDI  program  is  also  highly 
reliant  on  analytical  capability  consisting  of 
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supercomputers,  software  and  analysts,  in  order  to  test  and 
evaluate  a  number  of  potential  strategic  concepts  and 
systems  for  battle  management  and  command  and  control.  As 
of  early  1988,  there  was  difficulty  in  achieving  a  timely, 
secure  (i.e.,  cryptographic)  supercomputer  link  between 
three  key  analytical  sites  (70:1-5,9). 

National  Aero-Space  Plane 

The  National  Aero-Space  Plane  (NASP)  program  was 
established  in  December  1985  as  a  joint  Department  of 
Defense  and  NASA  program.  The  program  involves  such 
Department  of  Defense  agencies  as  the  Air  Force,  Navy, 

DARPA,  and  SDIO,  (60:1-10)  and  was  established  as  a 
"development  and  demonstration  program"  (60:22).  The  NASP 
has  been  tentatively  designated  as  the  X-30  since  it  is  "an 
experimental  vehicle  and  not  a  prototype  or  operational 
vehicle"  (60:27) . 

To  reduce  the  technical  risks  in  building  the 
prospective  X-30,  the  technical  community  has  identified 
"critical  or  enabling  technologies"  (60:10)  whose 
availability  and  maturity  should  be  monitored.  Such 
technologies  include  advanced  strength,  refractory  (i.e., 
high  melting  point)  materials.  To  gauge  the  technical  risk 
to  the  NASP,  engineers  have  compared  the  anticipated  fight 
trajectory  of  the  NASP  to  that  of  the  Space  Shuttle.  file 
the  Space  Shuttle  "reaches  orbit  very  quickly  in  an  almost 
vertical  flight  trajectory,  the  X-30  would  achieve  speeds 
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of  Mach  25  in  the  upper  atmosphere  before  [achieving] 
orbit"  (60:13).  Further,  the  X-30  would  take  off 
horizc  tally  and  be  essentially  air  breathing  while  the 
Space  Shuttle  does  neither.  However,  the  re-entry 
trajectory  for  the  X-30  is  expected  to  be  the  same  as  the 
Space  Shuttle  (19:67-68;  60:12-15). 

The  anticipated  X-30  trajectory  has  also  been  compared 
to  that  of  existing  aircraft.  Thus  it  is  expected  that  the 
"X— 30  must  be  designed  to  fly  ten  times  faster  and  higher 
than  existing  air-breathing  aircraft"  (60:12).  The  X-30  is 
therefore  expected  to  spur  greater  emphasis  on  hydrogen 
fuel  technology  (19:71;  60:12,15,40). 

According  to  NASP  and  NASA  scientists,  the  greatest 
technical  risk  connected  with  the  X-30  is  "the  development 
of  an  air-breathing  propulsion  system"  (60:25).  However, 
the  development  of  suitable  materials  and  the  integrating 
of  the  X-30's  major  subsystems  such  ac  the  thermal  control, 
structures,  propulsion  and  avionics,  also  provide 
formidable  technical  risks.  Indications  are  that  the  X-30 
design  and  development  will  require  significant 
computational  resources.  In  particular,  there  is  a  known 
requiiement  for  "computational  fluid  dynamics  to  predict 
the  aerodynamic,  thermal  and  propulsion  characteristics  at 
.  .  .  (Mach  8  to  25)"  (19:71;  60:25;  122:14). 

The  NASP  program  office  has  formulated  a  strategy  for 
reducing  the  technical  risks.  One  of  their  groundrules  is 
that  existing  national  facilities  (e.g.,  wind  tunnels  and 
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laboratories)  will  be  used  where  possible  to  preclude 
delays  due  to  construction  of  new  facilities  (60:26). 
Indeed,  it  has  been  contemplated  that  a  suitably 
instrumented  space  shuttle  could  be  used  to  aid  the  X-30 
design.  Another  groundrule  is  that  more  than  one  technical 
approach  will  be  pursued  for  high  risk  components  or 
systems  to  increase  the  chances  of  finding  appropriate 
solutions  in  a  timely  manner.  Consequently,  the  NASP 
program  intention  is  to  compete  a  number  cf  contractors 
with  the  expectation  that  each  contractor  will  pursue 
different — perhaps  even  highly  dif ferent — conceptual 
approaches.  Simultaneously,  the  NASP  program  intends  to 
promote  efforts  to  advance  those  technologies  that  are 
critical  to  the  X-30.  Additionally,  a  series  of 
milestones,  with  corresponding  design  reviews  and  key 
decision  points,  have  been  established  in  order  to 
facilitate  the  reduction  of  technical  risks  (60:26,35-40). 

The  plan  is  to  minimize  safety  risks  connected  with 
the  X-30  by  incorporating  the  appropriate  safety  features 
into  the  design  and  operation  of  the  vehicle.  For  example, 
the  X-30 1 s  propulsion  engine  will  have  at  least  two 
engines,  and  the  flight  control  system  is  planned  to  have 
four  backups.  Some  features  of  tne  vehicle's  operation 
would  also  contribute  to  safety.  The  flight  trajectory  is 
such  that  the  vehicle  would  largely  fly  above  adverse 
weather  (60:45-47).  In  the  event  of  an  aborted  mission, 
the  vehicle  will  be  able  to  maneuver  and  make  a  powered 


100 


SKIN  TEMPERATURES 
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Figure  6.  Technological  Trend  of  Increasing  Aircraft  Skin  Temperature  (140:68) 
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Figure  7.  Technological  Trend  of  Increasing  Engine  Temperature  (140:68) 


landing.  The  use  of  the  intended  hydrogen  fuel  has  been 
determined  to  pose  less  danger  than  conventional  fuels 
"since  its  ignition  temperature  is  1,065  degrees  Fahrenheit 
or  twice  that  of  aviation  grade  kerosene1'  (60:46). 

However,  the  technical  strategy  has  taken  into  account 
other  sources  of  technical  risk.  There  is  a  risk  of 
foreign  object  damage  to  the  X--30  since  "small  rocks  on  the 
runway,  hail,  ice,  rain,  or  even  space  debris"  (60:46) 
could  pose  considerable  danger  primarily  to  the  X-30's 
engines  and  skin.  Some  peripheral  technical  concerns  have 
also  been  considered  such  as  the  need  for  new  fuel 
processing  and  handling  procedures  (60:46-48,77,78). 

Summary  of  Case  Studies 

The  case  studies  reveal  that  the  sources  of  technical 
risk  vary  and  may  be  unanticipated.  For  the  F/A-18 
aircraft,  the  technical  risk  was  due  to  new  materials 
(composites)  that  were  used,  the  underestimated  need  for 
computer  memory  capacity,  and  certain  in-flight  failures 
that  led  to  crashes.  For  the  Chernobyl  Reactor,  human 
error  and  poor  design  introduced  technical  risk.  For 
airplane  development,  the  Wright  Brothers  addressed 
technical  risk  by  carefully  determining  the  cause  of 
Lil 1 ienthal ' s  fatal  crash  and  then  identifying  and 
concentrating  on  solving  what  they  determined  to  be  crucial 
for  achieving  successful  flight.  Specifically,  the  Wright 
Brothers  gave  the  solving  of  the  flight  equilibrium  problem 
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priority  over  the  development  of  a  suitable  engine  or 
propeller.  To  develop  the  rocket,  Robert  Goddard  worked  to 
first  understand  such  critical  rocket  components  as  rocket 
nozzles,  combustion  chamber  and  propellant  feed  system  and 
then  conducted  numerous  captive  and  flight  tests  of 
rockets.  The  Atomic  Bomb  development  required  not  only  the 
achievement  of  unprecedented  production  (i.e.,  separation) 
techniques  for  plutonium  and  uranium  isotopes,  but  also  new 
analytical  techniques  (e.g.,  Monte  Carlo  method)  and 
computational  tools  (i.e.,  the  electronic  digital 
computer) .  Laser  development  required  an  extrapolation 
from  maser  principles  and  the  overcoming  of  difficulties  in 
the  preparation  of  an  appropriate  optical  material. 

The  Apollo  11  landing  of  a  man  on  the  moon  required 
that  certain  formidable  prerequisites  be  satisfied  such  as 
the  development  of  a  high  thrust  vehicle  (the  Saturn  V) , 
selection  of  the  flight  path  and  moon  landing  sites,  and 
astronaut  training.  The  technical  risks  were  fundamentally 
overcome  in  stages  via  successive  space  flights  leading  up 
to  the  Apollo  11  moon  landing.  The  Apollo  13  Accident,  the 
Space  Shuttle  Challenger  Disaster,  the  Inertial  Upper  Stage 
Anomaly  and  the  AFOS  Software  problems  clearly  illustrate 
how  inadequately  managed  technical  risks  can  result  in 
significant  program  problems,  setbacks  or  disasters. 

The  case  studies  indicate  some  important  potential 
"unknown  unknowns"--f actors  which  constitute  unanticipated 
sources  of  technical  risk.  One  potential  unknown  unknown 
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is  the  need  for  one  or  more  secondary  development  efforts. 
For  example,  since  no  adequate  engine  or  propellor  was 
available  to  power  an  airplane  the  Wright  Brothers  had  to 
develop  both.  Similarly,  after  initial  launch  failures, 
Goddard  had  to  develop  specialized  high  pressure  pumps  to 
force  the  fuel  into  the  rocket  chamber.  The  Atomic  Bomb 
case  study  shows,  one  unknown  unknown  is  the  need  for 
relatively  unprecedented  analytical  techniques  and  tools. 
Figure  8  outlines  various  types  of  potential  "Unknown 
Unknowns. " 
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"UNKNOWN  UNKNOWNS" 
REVEALED  BY  THE  SELECTED 
CASE  STUDIES 


INADEQUATE  ANALYTICAL 
TECHNIQUES 


Atomic  Bomb  von 


Neumann  j  modeling  and 
simulations  required. 

Rockets  Reliability 
Modeling  required 


INADEQUATE  ANALYTICAL 
TOOLS 


Atomic  Bomb  Opportune 
emergence  of  the 
electicmc  computer 


CATASTROPHIC  FAILURE 


Apollo  1 : 

Fatal  launch  pad 

fire 

Apollo  13 

£  *  plosion  of 

oxygen  tank  m  space 

Chailenae 

r  Space  Shuttle 

Explosion 

during  launch. 

seven  astronauts  lulled 

UNDERESTIMATED 

CAPACITY 


Apcllc  1 1  Computer 


ove'loed  during  Moon 
landing 

AFOS  Software : 


Insufficient  computer 
memory 


EXCESSIVE  MODULE 
INTERDEPENDENCY 

AFOS  Software: 

Unnecessarily  complex 

software 

UNANTICIPATED 

SECONDARY 

DEVELOPMENT 

Airplane  WriQh*.  Brothers 
developed  lightweight 
engine  and  efficient 
P'CPCIIO'  for  plane 

Rocket.  Goddard 
deveioped  speoahzed 
rocket  fuel  pump 

FAILURE  INDUCED  by  USE 
ENVIRONMENT 

1 

1 

Project  Apono  Failure  of 

hydrogen  lines  during 

second  test  flight  ct 

Saturn  V  rocket 

INADEQUATE  DESIGN  FOR 
WORST  CASE  CONDITION 


Chernobyl  Nuclear 


Accioent  Reactor 


attained  critical  state 
extremely  rapidly  once 
deprived  of  cooling  water 
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IV.  Model  Development 


Analysis  Considerations 

Decision  making  must  often  be  made  in  the  presence  of 
risk.  To  reduce  technical  risk,  correct  and  adequate  risk 
assessments  must  be  made  by  technical  experts.  Then 
follow— on  risk  analyses  allow  effective  decisions  to  be 
made  by  identifying  those  alternatives  that  have  the 
highest  probability  of  success.  According  to  Klopp,  a 
successful  decision  (i.e.,  risk  analysis),  requires: 

1.  A  problem  to  be  solved 

2.  Viable  and  achievable  alternative  courses  of 
action 

3.  Information 

4.  A  decision  maker 

5..  A  decision  strategy.  (95:107) 

Timson  has  presented  the  general  model  for  system 
development  decision  making  shown  in  Figure  9.  Timson' s 
model  illustrates  the  potentially  cyclical  nature  of  system 
evaluation  and  revision.  Similarly,  General  Thurman 
emphasized  that  "...  the  planning  for  uncertainty  and 
risk  turns  out  to  be  a  dynamic  and  iterative  process" 
(143:15).  Further,  the  technical  and  performance  risk  is 
heightened  because  ".  .  .  complex  performance  requirements 

[of  multiple  subsystems  or  components]  .  .  .  must  interface 
perfectly.  .  .  ."  (143:12). 

Prevention .  Where  possible,  technical  risk  should  be 
prevented.  The  way  to  achieve  this  is  to  select  proven 


107 


rigure  9.  Model  of  System  Development  Decision- 

Making  (144:32) 
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techniques,  materials,  and  designs  which  involve  very 
little  uncertainty.  However,  a  Rand  report  has  stated  that 
Air  Force  missions  often  require  increasingly  higher 
technology  since  "the  newer  systems  must  perform  more 
functions  .  .  .  with  greater  precision  .  .  .  [with]  more 
integration  among  functions"  (129:23).  This  implies  that 
technical  risk  is  often  unavoidable. 

The  prevention  of  technical  risk  is  especially  crucial 
where  safety  is  concerned.  Accordingly,  MIL-STD— 882B 
System  Safety  Program  Requirements  states  that  the  first 
safety  precedence  is  to  eliminate  hazards  by  an  appropriate 
design.  Thus  the  first  design  priority  is,  if  at  all 
possible,  to  design  a  system  such  that  technical  risk  is 
eliminated.  However,  the  total  elimination  of  all 
technical  risk  in  a  particular  system  is  rarely  attainable. 
Therefore  the  second  design  priority  is  to  minimize 
technical  risk  by  appropriate  design  practices  (3:113; 

39:6) . 

Uncertainty  as  the  source  Risk.  Technical  risk  in 
complex  weapon  programs  stems  from  uncertainties  connected 
with  "...  inadequate  knowledge  of  the  basic  technology  or 
its  specific  implementation"  (143:12).  There  is  the 
initial  pitfall  that  the  actual  technical  problem  being 
evaluated  will  be  formulated  incorrectly  and  will  thereby 
result  in  the  right  answer  to  the  wrong  question  (126:15). 
For  a  risk  reduction  plan  to  be  effective,  it  must  lead  to 
successively  more  information  which  ultimately  identifies 
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and  reduces  both  the  anticipated  and  unanticipated 
technical  unknowns  connected  with  a  project.  Klopp  states 
that  there  is  a  "continuum  of  knowledge"  (80:148)  ranging 
between  having  no  information  arid  total  information  on  the 
internal  and  external  factors  that  govern  the  technical 
success  of  a  particular  system.  Clearly,  few  good 
decisions  will  be  made  when  there  is  no  information. 
Instead,  actions  are  often  undertaken  to  obtain  sufficient 
information  prior  to  a  decision.  Therefore,  the  most 
important  task  of  any  decision  maker  is  to  identify  where 
there  is  uncertainty  due  to  incomplete  information  and 
attempt  to  resolve  that  discrepancy  (80:148).  Kraemer  has 
proposed  the  risk  assessment  model  of  Figure  10  which 
results  in  a  quantified  risk  assessment  (i.e., 
probabilities) .  In  particular,  Kraemer  has  divided 
problems  into  "normal  risk"  and  "higher  risk"  in  his  model. 
Henley  and  Kumamoto  proposed  the  program  risk  assessment 
framework  of  Figure  11  which  begins  with  a  qualitative 
assessment  of  risk,  that  is  then  quantified  and  ultimately 
results  in  a  risk  management  strategy. 

Pitfalls .  Although  the  general  source  of  technical 
risk  is  uncertainty,  more  specific  factors  that  contribute 
to  technical  risk  are 

1.  Underestimation  of  the  degree  of  the  technological 
breakthrough  required  in  a  state-of-art  of  product 
development,  while  under  a  fixed  and  tight  time 
and  budget  constraint 

2.  Pushing  technology  too  fast 

3.  Lack  of  prototype  development 
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Ill 


RISK  ESTIMATION 


RISK  EVALUATION 


Figure  11.  Henley  and  Kumamoto’s  Risk  Assessment  Framework 

(89:14) 
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4.  Performance  requirements  beyond  state-of-the-art 

5.  Inadequate,  test  program 

6.  Major  design  or  scope  changes.  (132:35) 

These  stated  factors  can  become  pitfalls  if  they  ate 
overlooked.  The  case  studies  of  the  IUS  and  the  AFOS 
illustrate  that  technical  over-optimism  leads  to  an 
underestimation  of  the  technical  risks  involved.  To  ensure 
that  the  most  realistic  assessment  of  the  technical  risks 
are  obtained  as  soon  as  possible,  the  system  should  be 
prototyped  as  soon  as  practical.  Early  prototyping  of  the 
system  is  especially  important  if  the  in-house  experience 
level  of  the  technical  managers  and  designers  is  more 
theoretical  than  "hands-on".  Early  prototyping  has  been 
cited. 

State-of-the-Art .  Because  technological  risk  is 
determined,  in  large  part  by  the  state-of-the-art,  Rowe  and 
Somers  have  proposed  the  factors  cited  in  Figure  12  for 
evaluating  the  technological  state-of-the-art  (1^2).  In 
general,  the  higher  the  state-of-the-art  (i.e.,  the  newer 
the  technology) ,  the  greater  is  the  technological  risk 
involved.  Accordingly,  greater  emphasis  must  be  placed  on 
early  accurate  risk  assessment. 

The  technical  success  of  certain  development  efforts 
often  hinges  upon  advancerne*  cs  in  a  few  critical  areas. 

For  instance,  advancement  in  ".  .  .  propulsion  has  led  the 

way  for  all  major  advancements  in  flight  velocities" 
(5:327).  Therefore,  accurate  and  timely  identification 
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1.  Size— number  of  interrelated  components,  physical  volume 

2.  Complexity— difficulty  in  meeting  performance  requirement 

3.  Newness  of  technology— experimental  state  of  technology 

4.  Percent  proven  technology— degree  of  newness 

5.  Experience  in  the  field-work  on  similar  programs 

6.  Percent  new  components— test  and  evaluation  requirement- 

7.  Interdependency  of  subsystems— types  of  linkages 

8.  Degree  of  precision— quality  or  cleanliness  requirements 

9.  Special  resources— testing  or  tooling  requirements 

10.  Definitive  specifications— clarity  in  meeting  requirements 

11.  Design  flexibility — tolerance  level,  substitutes  available 

12.  Required  theoretical  analysis— need  to  support  proposed  design 

13.  Degree  difference  from  existing  technology— life  cycle  of 
technology 

14.  Available  knowledge  in  the  field— amount  of  experimentation 
required 

15.  Infra-structure  support  required— degree  of  dependency  on 
vendors 


Figure  12.  Factors  Determining  the 
State-of-the-Art  (132:40) 


of  those  high  risk  factors  and  components  which  are 
critical  to  correct  system  performance  is  mandatory  for 
reducing  technical  risk.  Once  identified,  these  critical 
technical  risk  areas  must  be  prioritized  by  experts  and 
emphasized  accordingly. 

However,  the  technical  success  of  one  system  may  also 
be  dependent  upon  the  technical  success  of  one  or  more 
separate,  but  allied,  system  or  systems.  For  example,  the 
GAO  had  determined  that  the  "Anti-Submarine  Warfare 
Standoff  Weapon  is  subject  to  increased  risk  due  to 
problems  in  a  related  program"  (58:8).  Technical  risk  also 
arises  due  to  excessive  interdependency  between  subsystems. 
The  AFOS  case  study  illustrated  that  if  software  modules 
are  made  highly  interdependent,  then  a  modification  to  one 
module  will  have  repercussions  to  other  software  module  and 
the  system  will  be  immensely  complex  to  modify, 
troubleshoot  or  update.  Therefore  the  interdependency  of 
subsystems  should  be  minimized  unless  absolutely  necessary. 

Expert  Resources.  To  avoid  or  reduce  technical  risk, 
it  is  critical  "that  the  proper  specialists  are  asked  the 
right  questions  at  the  right  time  and  that  their  advice  is 
heeded  when  given"  (137:83).  Technical  development  case 
histories  illustrate,  and  Shannon  stresses,  that  the 
violation  of  this  fundamental  concept  in  part  or  in  total, 
is  responsible  for  most  technical  failures. 

Timing .  Based  on  lessons  that  were  learned  in  DOD 
acquisition  it  is  well  established  that 
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.  .  .  the  concept  phase  is  the  critical  phase  for  risk 
and  uncertainty  analyses;  and  that  an  innovative 
tailored  approach  is  required  for  each  program  .  .  . 
the  analysis  at  concept  phase  establishes  the  basis 
for  the  rest  of  the  program  by  developing  system 
specifications  and  technology  base.  (143:21) 

Risk  assessment  and  analysis  is  needed  most  during  the 
concept  definition  phase — the  earliest  phase — of  a 
development  program,  because  it  is  here  that  technical 
uncertainty  is  greatest.  In  the  concept  definition  phase, 
there  are  usually  a  number  of  general  technical  approaches 
that  appear  viable.  The  correct  choice  from  among  these 
approaches,  along  with  successively  correct  choices  within 
the  selected  approaches  (e.g.,  design,  materials  and 
electronic  components)  is  significantly  aided  by  performing 
a  risk  assessment  (24:V-11;  143:21).  Further,  it  is 
critical  that  the  risk  assessment  be  available  for 
decisions  during  the  concept  studies  since  "by  the  end  of 
concept  studies,  70  percent  of  the  key  decisions  on  the 
weapon  system  have  been  made"  (128:138)  according  to  Lt  Gen 
Reynolds,  formerly  Vice  Commander  of  Air  Force  Logistics 
Command. 

Technical  experts  are  required  as  early  as  possible  to 
identify  and  select  those  design  alternatives  which  will 
preclude  as  much  technical  risk  as  possible.  Their 
expertise  then  becomes  critical  for  minimizing  risk  in  the 
design.  However,  because  a  "risk  assessment  is  the  first 
of  the  steps  in  a  risk  analysis"  (10:12),  the  risk 
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assessment  must  be  accurate  or  all  the  ensuing  effort  will 
be  invalid  to  some  degree. 

Planning  for  Technical  Contingencies .  The 
technological  uncertainty  inherent  in  a  new  technology 
development  program  means  that  the  "real  world  occurrence 
of  technological  breakthroughs,  and  catastrophes"  (144:93), 
particularly  the  latter,  must  be  realistically  considered. 
Specifically,  the  responsible  agency  must  ensure  that  " .  . 

.  historical  safety  data,  including  lessons-learned  from 
other  systems  are  considered  and  used"  (39:4)  to  improve 
the  contemplated  or  existing  design  of  the  system.  Such 
effective  design  techniques  such  as  simplicity,  modularity, 
redundancy  of  subsystems,  safety  and  design  margins,  etc., 
should  be  used  to  minimize  the  technical  risk  by  design. 

Among  other  things,  the  case  studies  illustrate  that 
especially  where  new  technology  is  involved,  established 
analytical  methods,  analytical  methods,  test  techniques, 
and  facilities  may  be  wholly  or  partially  in  adequate  or 
invalid.  Thus  the  success  of  the  Atomic  Bomb  development 
was  to  a  large  extent  dependent  on  Dr.  John  von  Neumann's 
modeling  of  critical  mass  phenomena  on  one  of  the  first 
electronic  computers.  It  is  probable  that  without  either 
the  contributions  of  Dr.  von  Neumann  or  the  opportune 
emergence  of  the  electronic  computer,  the  successful 
development  of  the  atomic  bomb  would  have  been  impossible, 
or  severely  delayed.  Similarly,  it  has  been  determined 
that  the  design  and  development  of  the  National  Aerospac  .* 
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Plane  (X-30)  would  require  the  use  of  computational  fluid 
dynamics  because  "wind  tunnels  are  of  no  use  in  modeling 
speeds  above  Mach  8"  (122:14).  In  general,  the  case 
studies  show  that  there  are  some  identifiable  categories  of 
technical  risk  that  are  usually  unanticipated.  Some  of 
these  unanticipated  categories  of  technical  risks — unknown 
unknowns — were  outlined  above  in  Figure  8. 

Analysis  of  Interviews.  The  interviews  were  conducted 
in  accordance  with  the  methodology  of  Chapter  II. 

Appendices  K  through  P  contains  transcripts  of  most 
interviews.  The  WRDC  interviewees  were  virtually  unanimous 
in  stating  that  technical  risk  reduction  is  not  the  first 
priority  of  the  laboratories.  Rather  the  primary  focus  in 
the  laboratories  is  in  exploring  the  technical 
possibilities.  For  example.  Dr.  Olsen,  Chief  Scientist  of 
the  wrdc  Flight  Dynamics  Laboratory,  stated  that  "Risk  is 
something  that  I  don't  think  that  we  [the  labs]  manage 
much.  We  generally  work  on  trying  to  prove  that  something 
is  feasible  and  can  be  done.  .  .  ."  (119).  However,  the 

scientists  stated  that  the  laboratories  are  implicitly 
involved  in  technical  risk  reduction  every  day  since  the 
labs  must  ultimately  transition  sufficiently  mature  (i.e., 
low  risk)  technologies  to  the  System  Program  Offices. 
According  to  Dr,  Keith  Richey,  the  Technical  Director  of 
WRDC,  the  labs  ".  .  .  make  a  conscious  decision  as  to  when 

to  turn  it  [the  technology]  loose"  (130)  since  they  must 
transition  the  technologies  relatively  fast  to  the  Air 
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Force  at  sufficiently  low  technical  risk.  According  to  Dr. 
Richey,  there  will  rarely  be  zero  risk  in  a  technology  that 
is  transitioned  to  the  SPOs  since  the  labs  cannot  examine 
all  possible  applications  of  a  particular  technology  (130). 

Dr.  Richey  stated  that  acquiring  " .  .  .as  much 
knowledge  as  you  can  have  on  the  physical  and  non-physical 
phenomena  that  you  are  dealing  with"  (130)  is  the  most 
important  technique  for  managing  technical  risk.  Similarly 
Dr.  Karris  Burte,  Chief  Scientist  of  the  WRDC  Materials 
Laboratory,  emphasized  that  "it  is  important  to  identify 
what  you  know  and  what  you  don't  know  and  that  what  is  what 
is  not  known  is  probably  the  most  important  source  of 
technical  risk"  (14).  According  to  Dr.  Arnold  Mayer,  Chief 
Assistant  for  Research  in  the  Vehicle  Subsystems  Division, 
analysis,  mathematical  and  physical  models,  tests,  and 
demonstrations,  are  used  in  the  iterative  process  of 
investigating  feasibility  and  that  this  equates  to 
investigating  technical  risk  (106).  Mr.  Jack  Cannon,  ASD's 
Technical  Director  for  Development  Planning,  stated  that 
"if  you  had  to  select  one,  then  certainly  prototype  testing 
is  the  way  you  reduce  risk"  (17).  Similarly,  Dr.  So  'ie 
Brown,  Chief  of  WRDCs  Technology  Assessment  Division, 
stated  that  a  suitable  demonstration  (e.g.,  flight 
demonstration)  is  essential  (13).  It  became  very  clear 
from  the  interviews  that  in  Dr.  Olsen's  words:  ".  .  .  risk 

in  the  laboratories  is  not  the  same  as  risk  in  a  SPO" 

(119) . 
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Model  Review  Comments 


The  preliminary  model  was  revised  to  incorporate 
comments  of  the  panel  of  experts.  Some  specific  written 
comments  from  their  reviews  may  be  found  in  Appendix  J. 
However,  many  of  the  comments  that  were  provided  were 
written  on  the  actual  copy  of  the  preliminary  model.  In 
addition,  information  obtained  during  the  interviewing  of 
the  experts  was  also  incorporated  into  the  model. 

The  model  had  initially  indicated  that  the  System 
Requirements  Specification  is  written  just  prior  to  the 
concept  exploration.  However,  Dr.  Squire  Brown's  comment 
was  that  it  would  most  likely  be  a  "needs"  statement  at 
that  stage.  He  also  indicated  in  his  review  that,  in  his 
words,  "there  may  be  an  intermediate  stage  before 
prototype — a  technical  demonstrator  like  the  X-29,  followed 
by  the  Advanced  Tactical  prototype."  The  model  was  revised 
to  incorporate  these  comments  (under  Stage  3.  and  Stage  7  of 
the  model,  respectively).  Because  Dr.  Brown  also  expressed 
that  some  transitions  in  the  model  were  unclear,  he  was 
provided  a  later  subsequent  version  of  the  model  for  an 
additional  review. 

Dr.  Jess  Riles1  first  model  review  comment  (Appendix 
I)  was  the  question  "Where  does  cost  enter  into  the  model?" 
Although  the  scope  of  this  research  (as  stated  under  Scope, 
Limitations,  and  Assumptions  in  Chapter  I)  does  not 
expressly  address  costs,  Dr.  Riles'  rationale  that 
"Technical  feasibility  may  be  demonstrated  but  may  not  be 


120 


affordable,"  is  compelling.  Therefore,  a  (prerequisite) 
block  was  added  under  Stage  1  of  the  model  that  requires 
that  one  "Identify  and  Confirm  Funding  Constraints."  His 
other  two  comments  were  in  accord  with  the  research 
groundrule  that  some  technical  feasibility  of  the 
development  program  in  question  has  been  demonstrated. 

After  Dr.  Squire  Brown's  second  review  of  the  evolving 
model,  he  had  some  additional  comments.  Specifically,  he 
recommended  that  the  design  priority  sequence  be  corrected 
to  show  that  the  first  design  priority  in  Stage  8  is  to 
achieve  the  primary  performance  objective  of  the  system  and 
only  then  is  the  rest  of  the  design  priority  sequence 
valid.  These  comments  were  discussed  by  phone  with  Dr. 
Brown  and  then  incorporated  into  the  model. 

Model  Presentation 

The  synthesized  risk  management  model  in  divided  into 
tan  stages  and  can  be  found  later  in  this  chapter  beginning 
on  page  131.  Citations  are  provided  to  document  the 
specific  source  whereby  each  respective  model  component  was 
derived.  Many  of  the  model  components  have  multiple 
citations  where  necessary  to  fully  document  che  model 
synthesis.  Some  blocks  in  the  model  indicate  pertinent 
technical  risk  management  examples.  These  specific 
examples  are  in  lower  case  letters  and  enclosed  within 
bracket;  all  other  citations  pertaining  to  the  block  are 
placed  outside  the  bracket.  For  example,  on  the  first  page 
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of  the  model,  the  Atomic  Bomb  is  listed  in  lower  case  as  an 
example  in  the  block  entitled  ''Identification  of  a  Real  or 
Potential  Threat."  The  citation  pertaining  to  the  Atom 
Bomb  example  is  placed  within  the  brackets. 

Logic  Symbols.  The  model  uses  or  and  and  logic 
symbols  or  "gates"  as  shown  in  Figure  13.  For  the  or  gate, 
a  correct  output  occurs  at  D  only  if  input  A  or  input  B  or 
input  C  is  present,  or  if  all  three  inputs  are  present 
simultaneously  as  inputs  to  the  or  gate. 

For  the  and  gate  in  Figure  13,  a  correct  output  is 
obtained  at  D  only  if  all  inputs  to  the  and  gate,  inputs  A 
and  B  and  C,  are  present.  If  any  of  the  indicated  inputs 
to  a  particular  AND  gate  are  missing  then  no  (valid)  output 
will  be  obtained  at  the  output  of  that  particular  and  gate. 
Thus,  the  and  gate  is  literally  being  used  as  a  "gate"  in 
the  model  that  ensures  that  the  correct  inputs  are  obtained 
before  any  output  is  passed  on  as  an  input  to  the  next 
succeeding  task. 

Main  Line.  A  key  feature  of  the  model  is  the  Main 
Line.  This  is  the  thick,  black  line  on  the  left  hand  side 
of  every  page  of  the  model.  The  risk  management  tasks  flow 
along  the  Main  Line  subject  to  the  fulfillment  of 
prerequisites.  These  prerequisites  are  indicated  by  blocks 
that  are  off  of  the  Main  Line  but  which  are  generally 
inputs  to  AND  gates  along  the  Main  Line.  For  ease  of 
reference,  all  blocks  along  the  Main  Line,  except  those  in 
Stage  8  of  the  model,  are  numbered  sequentially.  The  Main 
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Line  blocks  in  Stage  8  are  distinct  due  to  their  "priority" 
assignment. 

Design  Diamonds.  The  model  contains  four  diamond¬ 
shaped  blocks  which  indicate  that  specific  "Yes/No" 
question  is  asked  and  the  corresponding  path,  analogous  to 
a  decision,  is  then  followed  in  the  model. 

Requirement  Block.  The  model  actually  starts  with  the 
Requirement  Block.  The  OR  gate  that  precedes  the 
Requirement  Block  has  four  inputs:  (the  requirement  may  be 
mandated,  (2)  resulting  from  a  deficiency,  (3)  stem  from  a 
threat,  or  (4)  be  due  to  the  decision  to  pursue  a 
particularly  mission-enhancing  capability.  After  the 
Requirement  Block,  the  model  is  divided  into  Stage  1 
through  Stage  10. 

Stage  1.  After  the  requirement  has  been  specified 
and  authorized,  we  proceed  to  block  1  where  we  "Identify 
Appropriate  Technical  Disciplines  and  and  Experts  ..." 
that  are  pertinent  to  the  specific  requirement.  However, 
this  identification  of  experts  can  be  correct  only  if  the 
appropriate  representatives  from  the  using  organization, 
and  laboratory  experts,  and  individuals  with  previous 
experience,  that  are  pertinent  to  the  requirement,  have 
been  identified. 

Stage  2.  Similarly,  we  should  proceed  to  Stage  2, 
the  Initial  Study  by  Experts,  only  if  the  three  identified 
prerequisites  for  the  first  step  in  Stage  2  are  satisfied. 
During  this  stage  the  requirement  is  confirmed  and 
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communicated  to  the  experts)  along  with  any  salient 
priorities  within  the  requirement  that  the  using 
organization  has  identified.  This  way  when  the  cadre  of 
(national)  experts--of  the  correct  diversity,  depth,  and 
breadth  of  expertise — have  hopefully  studied  the 
requirement  somewhat  prior  to  their  assembly.  It  is 
therefore  possible  that  the  experts  may  arrive  at  the  first 
formal  assembly  having  already  identified  some  of  the  "Key 
Variables  Governing  Successful  Achievement  of  the 
Requirements"  or  with  an  understanding  of  various  technical 
tradeoffs  that  may  eventually  have  to  be  investigated  prior 
to  the  system  design  (Blocks  4  and  5) .  For  example,  for  a 
rocket  booster  as  the  Project  Apollo  case  study  made  clear, 
there  is  a  trade-off  between  the  weight  (e.g.,  structural 
strength)  and  the  payload  carrying  capability. 

At  Block  6,  the  broad  technical  objectives  of  the 
system  are  initially  specified.  These  objectives  should  be 
consistent  with  the  operational  requirements  from  the  using 
organization.  At  Block  7,  the  system's  technical 
objectives  are  initially  prioritized  although  this  may  have 
to  be  revised  based  on  results  of  forthcoming  analyses  or 
prototyping.  For  an  effective  prioritization,  one 
prerequisite  is  specific  previous  experience,  by  the 
individuals  doing  the  prioritization,  is  mandatory.  The 
other  immediate  prerequisite  shown  for  Block  7  is  a  network 
of  the  technical  tasks  (e.g.,  a  PERT  network)  would  be 
essential  for  showing  the  interrelationship  among  the  tasks 
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and  for  ultimately  determining  the  potential  impact  of  late 
completion  of  technical  tasks  on  the  entire  system 
development  schedule.  At  Block  8,  a  systems  requirements 
statement — much  more  general  than  a  specification — is 
written. 

Stage  3 .  During  Stage  3,  the  concept  exploration 
and  technical  risk  assessments  are  conducted  as  Blocks  9 
and  10  show.  The  pitfall  identified  in  conjunction  with 
Block  9  is  that  technical  overoptimism  must  be  guarded 
against  to  ensure  that  no  potential  weaknesses  or  problems 
with  a  contemplated  technical  concept  are  overlooked. 

Eight  prerequisite  tasks  are  shown  before  the  Block  10 
Technical  risk  Assessment  task  can  be  considered  completed. 
For  instance,  the  "level  of  the  technical  expertise  that  is 
needed  vs  available"  must  be  evaluated.  In  general,  the 
technical  risk  assessment  will  have  to  be  done  for  each 
concept  that  is  examined.  In  Block  11,  the  technical  risks 
corresponding  to  each  concept  are  prioritized  (i.e.,  rank 
ordered)  . 

Stage  4 .  In  Stage  4,  specifically  Block  12,  a 
concerted  effort  is  made  to  identify  and  plan  to  resolve 
the  important  unknowns  that  have  a  bearing  on  the  outcome 
of  the  system  development  effort.  In  order  to  achieve  the 
Block  12  task,  one  prerequisite  is  for  resolving 
information  uncertainty  is  to  "Identify  Relevant  National 
Superlabs,  Expertise,  .  .  .  ."  and  existing  simulations  or 

software  that  can  help  to  resolve  any  information 
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uncertainty  relevant  to  the  technical  effort.  This  stage 
is  also  important  because  it  may  prevent  unnecessary 
duplication  of  tasks  that  have  already  been  done  elsewhere. 

Stage  5.  In  Stage  5,  Block  13,  there  is  a 
concentrated  effort  to  acknowledge  a  plan  for  "what-ifs" — 
to  consider  courses  of  action  prior  to  the  actual 
occurrence  of  technical  problems.  In  Block  14  through  16, 
criteria  for  evaluating  candidate  approaches  are 
identified.  The  approaches  (Alternatives)  are  then 
analyzed  against  the  criteria  and  those  approaches  that 
meet  or  exceed  the  criteria  are  selected  as  candidate 
technical  approaches.  Finally,  in  Block  17,  the 
alternatives  which  survive  the  screening  are  tentatively 
rank  ordered  for  the  most  rigorous  study  in  Stage  6. 

Stage  6.  In  Stage  6  there  is  an  intensive  effort  to 
ensure  that  a  rapid  leaning  curve  is  achieved  with  respect 
to  the  surviving  technical  approaches  and  the  potential 
technical  risks.  Per  Block  18,  the  candidate  concepts  are 
investigated  in  detail  with  respect  to  the  prerequisites 
identified:  "Analytical  and  Simulation  Tools,  Math 

modeling,  etc."  In  Block  19,  in-depth  simulation  or 
physical  models  are  used.  For  adequate  risk  reduction,  one 
cannot  proceed  past  Block  19  unless  the  applicable  "worst 
case"  parameter  values  as  well  as  "most  likely"  parameter 
values  have  been  identified  for  the  design  under  study. 
Further,  it  must  be  ascertained  that  the  simulation  is 
representative  of  "real  world"  conditions  in  order  for  the 


127 


simulation  output  to  be  of  value.  As  part  of  the  learning 
curve  process,  the  organization's  technical  knowledge  must 
be  continually  enriched  (Block  20) .  This  enrichment  can 
take  the  form  of  recruiting  "fresh”  technical  personnel 
from  time-to-time,  training,  and  attendance  at  appropriate 
technical  seminars. 

Stage  7 .  In  this  stage  the  basic  system  design 
corresponding  to  each  candidate  approach  is  outlined  (Block 
22)  and  key  technical  parameters  for  measuring  technical 
progress  of  the  prospective  system  are  tentatively 
identified  (Block  23) .  Some  experimental  prototyping  of 
some  components  or  subsystems  that  are  anticipated  to  be 
high  risk  (Block  23)  is  pursued  in  this  stage.  Assuming 
that  the  indicated  prerequisites  are  met,  one  or  more 
Preliminary  Design  Reviews  (Block  24)  are  held  to  select 
the  primary  technical  approach  for  the  system  and/or  for 
the  system's  key  subsystems  (Block  25).  Block  26  is  a 
decision  diamond  regarding  whether  or  not  the  selected 
primary  approach  should  be  pursued  solely.  If  the 
technical  approach  selected  is  not  relatively  old  then  we 
come  off  the  "No"  side  of  the  diamond  and  pursue  two  or 
more  approaches  in  parallel.  If  the  technical  approach  is 
old  than  we  come  off  the  "Yes"  side  of  the  diamond.  In 
either  case  we  go  through  the  OR  gate  to  a  decision  diamond 
regarding  the  experience  of  the  personnel  (Block  28)  in  the 
selected  approach  or  approaches.  If  the  personnel  are  not 
experienced,  then  we  rush  to  Stage  8,  Design  of  Prototype." 
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block  29  likewise  requires  a  decision  on  the  technical  and 
operational  familiarity  with  the  environments  within  which 
the  system  will  operate.  This  results  in  selection  of  the 
Block  30A  or  Block  30B  tasks  to  determine  how  the  system 
will  function  in  the  use  environment.  Block  31  involves 
the  formation  of  specific  strategy  to  minimize  the  risks 
involved  with  eventually  achieving  a  successfully 
functioning  prototype.  Such  a  strategy  can  then  be  applied 
to  actual  prototype  design  (under  Stage  8).  At  Block  32, 
experimental  prototypes  can  be  used  in  early  field  tests  in 
order  to  obtain  an  early  evaluation  of  suitability  to  the 
operational  environment. 

Stage  8.  This  stage  involves  the  actual  design 
and  building  of  the  prototype  system.  Stage  8  is  divided 
into  nine  (9)  design  priorities  ranging  from  designing  to 
meet  or  exceed  the  primary  performance  objectives  to 
designing  for  easy  modification  at  subsystems.  Thus,  the 
highest  design  priority  is  Design  Priority  1  and  the  lowest 
design  priority  is  Design  Priority  9.  Collectively,  these 
design  priorities  ensure  that  the  higher  priority  design 
task  is  achieved  before  a  lesser  task  is  given  emphasis. 

In  practice  these  design  priorities  may  be  pursued  somewhat 
simultaneously  if  the  indicated  prerequisites  of  each  have 
been  met. 

Stage  9 .  Once  the  prototype  has  been  designed  and 
built  ,  a  technical  risk  evaluation  of  the  actual  system  is 
made.  The  potential  hazards  that  may  arise  from  the  system 
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(e.g.,  explosion,  fire)  are  identified  (Block  33).  The 
Boeing  checklist  in  Figure  4  can  be  used  for  this.  The 
prototype  must  then  be  evaluated  for  possible  failure  modes 
and  the  consequences  of  those  failures  (Blocks  34  through 
38) .  The  failure  mode  analyses  along  with  information 
derived  from  testing,  will  eventually  result  in  the 
implementation  of  design  improvements.  Then  more  Critical 
Design  Reviews  can  be  held  to  evaluate  the  technical 
progress  and  to  decide  on  necessary  design  changes  for 
subsequent  (e.g.,  more  mature  versions)  prototypes. 

Because  the  Critical  Design  Review  occurs  after  a  prototype 
has  been  designed  and  built  instead  of  before,  more 
technical  information  upon  which  to  base  a  decision 
regarding  necessary  design  modifications. 

Stage  10.  Detailed  testing  of  the  prototype  (or  a 
later  version)  is  conducted  in  Stage  10  of  the  risk 
management  model.  "Specific  Test  Objectives"  and 
"pass/fail  criteria,"  along  with  the  other  items  shown,  are 
prerequisites  for  effective  testing  (block  41) .  At  the 
decision  diamond  (Block  42)  technical  experts  must  be 
consulted  (Block  43A)  if  there  is  a  test  failure  and  design 
changes  must  be  implemented  (Block  44) .  After  the  design 
changes  there  will  be  a  retest.  If  the  system  passes,  no 
further  design  changes  are  implemented  (Block  43)  and  the 
readiness  for  production  of  the  system  is  evaluated  in 
accordance  with  Air  Force  System  Command  Regulation  84-2  on 
Production  Readiness  Reviews. 
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MODEL  FOR  TECHNICAL  RISK  MANAGEMENT 
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9  CONDUCT  FEASIBILITY/CONCEPT  STUDIES 
(143:15) 


AVOIO  PITFAU  OF  TECHNICAL  OVER- 
OPTIMISM  REGARDING  THE  EASE  OF 
APPLYING  NEW  OR  MODIFYING 
ESTABLISHED  TECHNOLOGY 
(28:24;  48:128;  49:61) 


EVALUATE:  LEVEL  OF  TECHNICAL 
EXPERTISE  NEEDED  VS  AVAILABLE 
(132:40;  143  12) 


DEFICIENCIES.LIMITATIONS  OF  EXISTING 
TEST  FACILITIES  AND  EQUIPMENT,  NEW 
TEST  RESOURCES  REQUIRED 
(5:8;  122:14) 


IDENTIFY  DESIGN  OR  FABRICATION  ITEMS 
NOT  REDUCED  TO  PRACTICE 
(9:256;  97:6,7) 


EVALUATE  STATE-OF-THE  ART  PER  FIGURE 
12  (28:27;  82.261.  132.40) 


ASSESS  FABRICATION  AND 
MANUFACTURING  RISK  PER  DOD  MANUAL 
4245. 7-M  (20  2;  36) 
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10  CONDUCT  TECHNICAL  RI5K  ASSESSMENT  DURING  CONCEPT 
STUDY  PHASE 
(85:217;  143:17.21) 


11  PRIORITIZE  ELEMENTS  OF  TECHNICAL  RISK 
(28:24.  108:26) 


STAGE  4:  AOORESS  INFORMATION  RISK 


IDENTIFY  RELEVANT  NATIONAL  SUPER¬ 
LABS,  EXPERTISE,  SIMULATIONS 
(80.181) 


IDENTIFY  CHITICAL  FUNCTIONS  REQUIRING 
EARLY  PROTOTYPING 
(25:Sec  2,6;  108:26) 


12  SPECIFY  PLAN  TO  OBTAIN  ADDITIONAL  INFORMATION 
CORRESPONDING  TO  EACH  ITEM  OF  TECHNICAL  UNCERTAINTY 
(30:15;  150  43-47) 
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STAGE  5:  INITIAL  TECHNICAL  CONTINGENCY 
PLANNING 


I 


16.  iDENTIfr  PO  rtNTl  ALLY  VIABLE 
TECHNICAL  ALTERNATIVES  /  CONCEPTS 
BASED  ON  SCREENING  CRITERIA 
(45.324J25) 


AVOID  PITFALL  OF  UNREALISTIC. 
UNREPRESENTATIVE  SIMULATION  OF  THREAT. 
ENVIRONMENT 

(51:5.  52:6;  71:3,  73:13; 112:3) 
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23.  BEGIN  EXPERIMENTAL  (PARALLEL) 
PROTOTYPING  OF  COMPONENTS  OR 
SUBSYSTEMS  THATHAVE  THE  MOST 
TECHNICAL  RISK 


24  PRELIMINARY  DESIGN  REVIEW 
PER  APPENDIX  O  OF  MIL-STD-1521 
(28:33-52) 


ATTENDANCE  BY  COGNIZANT  USER 
REPRESENTATIVES  (30  Sec  XV.  2;  89  19.  24) 


RESULTS  OF  INITIAL  ANALYSES.  STUOlCS. 
UPDATED  TECHNICAL  RISK  ASSESSMENT 
(28:33:  143:12) 


SPECIFICATIONS 

(28:33) 


EXPERIMENTAL  PROTOTYPES.  MODELS 
(28:33) 


INTEGRATION .  INTERFACE  REQUIREMENTS 
(112  29;  143:12) 


EARLY  TEST  PLANNING 
(32:2) 


MOST  PROVEN  TECHNOLOGY  BASE 
(19:52,54;  97:1;  133:13; 

140:68;  143:13) 


LEAST  MANUFACTURING  /  FABRICATION 
RISK  PER  OOO  MANUAL  424S  7-M(3G) 


MINIMUM  SAFETY  RISK  PER  MIL-STD-882 
(39) 
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29. 


141 
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STAGE  8: 


1st  Design  Prionty: 

DESIGN  TO  MEET  OR  EXCEED  PRIMARY 
PERFORMANCE  OBJECTIVES  (13) 


OF  PROTOTYPE 
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FIXED  OR  AUTOMATED  CORRECTION 

DEVICES 

(39  6) 


MANUAL  CONTROL  OVERRIDE  (e  g  , 
during  Apollo  11  landing  Nod  Armstrong 
manually  overrode  the  computer  to  avoid 
landing  on  site  strewn  with  boulders 
(91:54;  116:767)1 


EMERGENCY  PROCEDURES 
(39.6) 


TRAINING  FOR  EMERGENCIES 
(150:44) 


TRAINING  EQUIPMENT  (e  g  ,  moon 
landing  (light  simulators  built  and  usea 
on  Earth  (150:44)1 


CAPABiLITY  TO  SIMULATE  MALFUNCTION 
OR  CHANGES  FOR  STUDY  (During  Apollo 
13  inflight  accident,  ground  simulations 
identified  ways  to  power  down  jnJ  save 
energy  (90  60;  91:56 ;  112.9) 


6th  Design  Prior  its 


QUICK  CORRECTION:  OESlGN  TO 
PROVIOE  OPTIONS  WHICH  ENABLE 
QUICK  RECOVERY  FROM 
MALFUNCTION  (62:2.3,33;  91  54) 
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7th  Oesicin  Priority: 

OESIGN  FOR  ACCURATE,  RAPIO 
IDENTIFICATION  OF  FAILED 
SUBSYSTEM 
(29  35;  129:19) 


8th  Design  Priority 


OESIGN  FOR  EASY  REPAIR  OF 
FAILED  SUBSYSTEM 
{29  43-45) 
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MODULARITY 

(29:43-45) 


L 


J 


STANDARD  PARTS  AND  MATERIALS 
(NON-EXOTIC)  (28:24) 


LOW  INTERDEPENDENCY  OF  MODULES 

SUBSYSTEMS 

(65:31) 


AVOID  PITFALL  OF  SOFTWARE  OR 
SYSTEM  DOCUMENTATION  WHICH  OOES 
NOT  REFLECT  CHANGES  IN  A  TIMELY 
MANNER  (65  35) 


STAGE  9:  POST-DESIGN  RISK  ANALYSES 
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TEST 


SPECIFIC  TEST  OBJECTIVES.  (125:3) 
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V.  Conclusions  and  Recommendations 


Overview 

Inadequately  managed  technical  risks  have  resulted  in 
setbacks,  failures  and  operational  disasters  in  Department 
of  Defense  programs.  Therefore,  the  purpose  of  this 
research  was  to  synthesize  a  model  that  epitomizes  a 
strategy  for  the  management  of  technical  risk.  The  model 
was  synthesized  using  the  three-pronged  effort  of:  (1)  a 
literature  search  and  review  to  determine  what  previous 
work  was  done  in  risk  assessment  and  risk  management,  (2) 
case  studies  of  historical,  contemporary  and  prospective 
new  technology  development  programs,  and  (3)  interviews 
predominately  with  Chief  Scientists  at  the  Wright  Research 
and  Development  Center  (WRDC)  at  Wright-Patterson  Air  Force 
Base.  The  model  validation  was  via  reviews  of  the  model 
that  were  made  by  the  stated  interviewees.  If  the  model  is 
used  as  a  technical  management  guide  and  decision  aid  by 
individual  program  or  project  managers  at  all  levels, 
collective  marked  improvement  in  the  technical  risk 
management  throughout  the  Department  of  Defense  may  be 
achieved . 

Conclusions 

The  research  reveals  that  the  management  of  technical 
risk  is  essentially  a  sequential,  cumulative,  repercussive 
and  iterative  activity.  Technical  risk  management  is 
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basically  sequential  because  the  start  of  one  particular 
task  often  requires  the  successful  completion  of  one  or 
more  preceding  tasks;  one  task  often  requires  inputs  from 
one  or  more  previous  tasks.  Technical  risk  is  cumulative 
because  failure  to  adequately  fulfill  the  prerequisite 
tasks  contributes  to  the  technical  risk  for  all  succeeding 
tasks.  For  example,  the  act  of  selecting  an  inferior  or 
uncertain  design  approach  may  thereafter  incur  significant, 
risk  to  successful  system  development  regardless  of  whether 
or  not  all  succeeding  tasks  are  performed  in  exemplary 
fashion.  Technical  risk  management  is  repercussive  because 
changes  to  a  particular  component  in  a  system  may  cause 
undesirable,  or  even  unanticipated,  technical  ripples 
throughout  the  entire  system.  Therefore,  to  minimize 
system  complexity  and  risk,  it  is  important  to  understand, 
and  if  possible,  limit  the  interdependence  of  modules  and 
subsystems  within  a  system. 

Finally,  technical  risk  management  is  iterative 
because  as  the  technical  effort  continues,  knowledge  gained 
about  system  deficiencies  or  potential  for  improvement 
results  in  changes  to  the  system.  Since  changes  occur 
during  the  development  of  virtually  all  systems,  the 
technical  approach  must  be  flexible  enough  to  accommodate 
system • revisions  indicated  by  analysis,  test,  production, 
and  operation. 

According  to  Mr.  C.  J.  Cosenza,  Chief  of  the  Advanced 
Development  Division  of  WRDC's  Technology  Exploitation 
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Directorate,  technical  risk  is  usually  managed  in  stages  so 
that  the  risk  is  systematically  less  in  successive  stages 
(22) .  However,  he  emphasized  that  where  time  is  the  major 
factor,  such  as  when  a  system  is  urgently  needed  to  counter 
a  known  threat,  then  multiple  tasks  must  proceed  in 
parallel  since  technical  risk  considerations  and  cost 
become  secondary  to  the  schedule  in  that  case  (22). 

Model  Value/Validation 

The  technical  risk  model  has  been  reviev-ad  by 
cognizant  technical  experts  of  WRDC.  Appendix  J  contains 
written  comments  that  were  provided,  on  the  preliminary 
model.  Mr.  Jack  Cannon,  Technical  Director  of  the 
Development  Planning  in  the  ASD  Office  of  Plans,  said  that 
the  risk  management  model  is  valuable  to  the  program 
manager  who  skips  some  of  the  intermediate  steps  to  show 
him  the  “management  risk  he  is  taking  ...  by  treating  as 
unnecessary  certain  steps"  (Appendix  J) .  That  is,  the 
model  prevents  the  program  manager  from  unknowingly 
skipping  steps  or  being  unmindful  of  the  potential 
technical  future  risks  he  may  be  incurring  to  the 
development  of  a  system  by  doing  so.  Dr.  Jim  Olsen,  Chief 
Scientist  of  the  WRDC  Flight  Dynamics  Laboratory  wrote  that 
"the  logic  (in  the  model]  is  impeccable."  He  stated  that 
the  important  technical  "question/answers  are  not 
frequently  addressed,  or  are  addressed  in  an  uncoordinated 
way"  during  systems  development.  Therefore,  the  technical 
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risk  management  odel  may  ensure  that  the  crucial  questions 
and  answers  have  been  systematically  asked  and  obtained, 
respectively,  prior  to  each  important  system  decision  to 
achieve  technical  risk  reduction.  The  model  was  developed 
based  upon  inputs  revised  from  an  exclusive  literature 
review,  case  analyses,  and  interviews  with  experts. 
Information  was  then  synthesized  into  a  realistic  and 
workable  out  put  the  mode. 

After  his  second  review  of  the  evolving  model.  Dr. 
Brown  remarked  that  the  model  is  "an  interesting  piece  of 
work"  and  that  it  is  in  accord  with  his  experience  in  the 
area  of  technical  risk  assessment/management.  Mr.  Cannon's 
comment  that  "unknown  unknowns"  (i.e.,  unanticipated  events 
that  cause  technical  risk)  cannot  actually  be  evaluated. 
Therefore,  the  prerequisite  regarding  "unknown  unknowns" 
(near  Block  13)  was  clarified  to  mean  the  "potential 
unknown  unknowns"  that  were  derived  from  the  case 
histories . 

Recommendations  for  Further  Research 

Although  the  technical  risk  management  model  has  been 
validated  by  a  group  of  technical  experts,  it  now  requires 
formal  testing  in  System  Program  Offices. 

DSMC  Evaluation.  The  model  should  be  provided  to  the 
Defense  System  Management  College  (DSMC)  for  their 
evaluation  and  possible  incorporation  into  acquisition  risk 
management  strategy.  Speci f ical ly ,  the  model  may  be  very 
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beneficial  when  used  in  conjunction  with  KIL-STD-1521B — 
essentially  an  effective  technical  checklist.  Recently, 
the  Willoughby  templates  (36)  have  been  drafted  as  part  of 
a  strategy  to  attempt  to  decrease  the  Department  of  Defense 
risks  in  transitioning  programs  from  development  to 
production.  The  model  herein  encompasses  technical  risk  in 
general  and  references  the  Willoughby  templates  via 
reference  (36).  The  model  may  therefore  provide  a  strong 
foundation  for  a  general  comprehensive  technical  risk 
management  strategy. 

Subjective  Evaluation.  One  method  of  doing  this  would 
be  to  provide  the  model  as  a  technical  risk  management 
strategy  and  decision  aid  to  ten  (10)  Department  of  Defense 
program  managers  (at  the  Captain  to  Lieutenant  Colonel 
levels  and  civilian  equivalents) .  This  group  will  be 
designated  as  the  "model  group."  After  a  definite  time 
interval  (e.g.,  18  months)  the  group  with  the  technical 
risk  model  should  be  asked  to  provide  specific  comments  as 
to  the  utility  benefit  or  shortcomings  of  the  model. 

Testing.  For  actual  testing,  a  control  group  of 
program  managers,  that  will  not  receive  the  model,  will  be 
identified.  The  objective  of  the  trial  would  be  to 
determine  whether  the  programs  being  managed  by  individuals 
in  the  model  group  achieve  a  superior  technical  status  an 
indication  of  better  technical  risk  management--than  the 
control  group.  Realistically,  since  the  required  period 
over  which  to  evaluate  technical  progress  could  be  long, 
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the  comparison  can  be  made  by  evaluating  whether  the  model 
the  control  group  are  practicing  a  better  technical  risk 
assessment  and  management  based  on  predetermined  criteria. 

Specifically,  at  the  end  of  the  trial,  participants 
would  complete  a  survey  questionnaire  (Likert  scale)  which 
would  attempt  to  evaluate  to  what  extent  they  are  following 
effective  technical  risk  management  techniques.  For 
instance,  they  would  be  asked  to  respond  on  a  Likert  scale 
to  what  extent  they  have  ensured  or  are  personally  aware 
that  the  following  have  been  (or  are  being  done)  in  their 
program: 

1.  Specific  laboratory  experts  that  could  help  their 
program  in  a  consulting  capacity  in  the  event  of  technical 
problems,  have  been  identified. 

2.  Technical  program  risks  have  been  identified. 

3.  Technical  program  risks  have  been  prioritized. 

4.  Interface  requirements  have  been  (are  being) 
specifically  defined. 

5.  Technical  personnel  connected  with  the  program 
attend  professional  seminars  and  obtain  suitable  training. 

6.  There  is  a  capability  to  simulate  or  reproduce 
system  malfunctions  for  study. 

7.  Personnel  from  the  using  organization  who  can 
provide  more  inf ormat ion- or  details  on  the  use  environment 
and  the  specific  employment  of  the  system  have  been 
specifically  identified. 
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8.  Exotic  materials,  parts  and  processes  of  the 
system  have  been  specifically  identified. 

9.  Backup  options  for  exotic  materials,  parts  and 
processes  have  been  identified. 

10.  The  specific  worst  case  locales  where  the  system 

\ 

man  be  used  (e.g.,  sandy,  dusty,  cold,  hot,  at  altitude  or 
a  combination)  have  been  specifically  identified. 

The  ten  criteria  just  stated  are  a  partial  list  of  the 
criteria  that  could  be  used  as  indicators  of  how  well  the 
technical  risks  of  a  program  are  being  managed.  Although 
the  responses  to  the  questionnaire  would  not  guarantee  that 
a  particular  program  manager  is  in  fact  managing  technical 
risk  as  indicated  by  his/her  responses,  the  responses 
should  generally  be  a  good  measure  of  risk  management.  If 
the  model  is  of  high  utility  then  theoretically  the  model 
group  should  score  higher. 

Given  the  variation  in  program  type,  maturity,  and 
external  influences  (e.g.,  budget,  schedule  changes)  in  the 
Department  of  Defense  it  may  be  necessary  to  select 
programs  which  are  at  or  near  the  same  Department  of 
Defense  acquisition  milestone,  or  to  ensure  that  the  model 
and  control  groups  have  the  same  mix  of  programs  at  the 
various  milestones. 

Management  Implications  to  POD 

To  successfully  achieve  the  objectives  of  a  particular 
Department  of  Defense  weapon  development  program,  the 
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program  manager  must  effectively  manage  the  program's  risk. 
Since  technical  risk  often  has  a  direct  bearing  on  the  risk 
of  failing  to  achieve  both  a  target  cost  and  schedule,  it 
is  essential  that  an  effective  strategy  be  formulated  and 
implemented  for  managing  risk.  The  importance  of  strategy 
has  been  previously  discussed  in  Chapter  I  (Justification) . 
However,  a  particularly  noteworthy  example  of  the  potential 
advances  that  can  be  achieved  by  implementing  an  effective 
strategy  is  the  emergence  of  Japan  as  a  dominant  of  Japan's 
defeat  in  World  War  II.  The  Japanese  have  "attributed  much 
of  their  success  to  the  teachings  of  such  Americans  as  (Dr. 
W.  Edwards]  Deming"  (136:541)  in  the  areas  of  quality  and 
industrial  management.  The  nation  that  can  more  ably 
manage  technical  risk  is  likely  to  have  decided  defense  and 
industrial  advantages  over  nations  with  lesser  technical 
risk  management  skills.  With  a  superior  technical 
management  strategy,  a  greater  percentage  of  defense 
programs  may  be  done  correctly  "the  first  time,"  and  within 
budget  and  schedule.  Further,  a  period  of  diminishing 
Department  of  Defense  resources  and  a  public  that  is 
increasingly  intolerant  of  even  minor  defense  acquisition 
fiascos,  may  require  that  an  improved  technical  management 
strategy  be  implemented. 

It  is  likely  that  no  general  officer  would  propose 
that  a  war  be  fought  without  a  potentially  effective  combat 
strategy  which  pervades  even  the  lowest  echelons.  By 
analogy,  defense  weapons  development  and  acquisition  could 
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benefit  from  a  pervasive  technical  risk  management  strategy 
that  is  available  even  to  lower  level  program  and  project 
managers.  Such  a  deliberate  strategy  could  supplant 
marginally  effective — and  perhaps  impromptu — technical  risk 
management  approaches  that  may  be  in  use  by  some  Department 
of  Defense  program  managers  in  the  day-to-day  management  at 
the  lower  and  middle  level  (officer)  echelons.  A  codified 
technical  risk  management  model  is  likely  to  enhance 
communication  and  synchronize  the  efforts  of  managers  and 
engineers  at  all  levels  since  that  model  would  provide  a 
common  reference  point. 

Ideally  then,  the  model  that  resulted  from  this 
research  could  be  provided  as  a  reference  to  every  program 
manager  at  almost  all  levels.  The  technical  risk 
management  improvements  which  the  model  may  facilitate  for 
each  individual  program  manager  could  result,  collectively, 
in  a  quantum  improvement  of  technical  risk  management  in 
weapons  acquisition  throughout  the  Department  of  Defense. 
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Augu*t  2uo ,  1819 


T.J>.  Roo««».lt, 

frooldaet  of  th*  Unit*!  SttUi, 

Thltt  Hout* 

Washington,  D.C. 

Sir  i  . 

Soot  recant  work  by  E.ftnl  and  1.  Sxllard,  which  ho*  b*«o  sow. 
aunlcoltO  (e  st  In  wnnuacrlpt.  load*  wo  to  txpact  thol  tho  alaaant  uran- 
lua  nay  bt  turned  Into  o  now  wnd  laportant  source  of  tntrgy  in  th*  Iw- 
todtoto  future.  Ctrtoln  oopocto  of  tho  oltuotlon  which  ho*  ariooo  oooa 
to  coll  for  t.otchfulnott  and.  If  nocotaary,  quick  octtoo  on  th*  port 
or  th*  Odnlnltirot ion.  1  boll***  thtrtfor*  that  It  1*  ay  duty  to  bring 
to  your  attention  th*  fallowing  foot*  and  rocernondatlonoi 

In  th*  court*  of  th*  loot  four  wentht  it  hot  b««n  cod •  probobi*  - 
through  th*  work  of  Jellot  In  franc*  at  wall  o*  ?*ral  ond  Sellar*  in 
*o«nc»  -  that  it  may  btcoa*  poitlbl*  to  oat  up  o  nuclear  chain  raoction 
in  a  lorgt  not*  of  uronluw.by  which  *ott  aoounla  of  pow*r  and  lorg*  quant* 
ltlat  of  non  rodiiua*llk«  *l*a*ot*  would  b*  gonarattd.  Vow  it  op  poor* 
olaot t  ctrtoln  that  thl*  could  b«  achieved  In  th*  twntdlat*  futur*. 

Thi*  n«o  phanonanon  would  alao  l«ad  to  th*  conatructloo  of  beabt. 
and  it  |*  cone* l*abl*  .  though  (Mich  1*»*  oortalo  -  that  *str*aaly  power* 
ful  towb*  of  a  non  typ*  way  thus  bt  'construct**.  A  single  bmb  of  thl* 
typo.  earn**  by  boat  and  oaplod*d  la  a  port,  nl*ht  **ry  wwll  dtotroy 
th*  nbol*  port  togsthor  with  tow*  of  th*  surrounding  territory.  Howovor. 


1  understand  that  Gtraany  ha*  actually  stoppod  th*  ool*  of  uronlua 
fron  th*  Ciochotlsvaklan  win**  which  ah*  ha*  takta  o**r.  That  *h«  should 
have  t*»«n  tuch  aorly  action  night  ptrhop*  b<  und*r*to«d  on  th*  ground 
that  th*  ton  of  th*  G* man  Ond«r*S*cr*lory  of  Slat*.  ran  Tfltoackcr.  1* 
attach**  to  th*  Kalasr-'nihslw-lnotltut  la  larltn  wh*r*  tow*  of  th* 
aaarlcaa  work  on  uraalua  la  now  being  rop«al«d. 

Your*  T«ry  truly. 

(aibtrt  xiooioia) 


ftam  Fiankhn  D.  Haemtli  Ltbiarj 
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Appendix  B:  Interview  Questions 


1.  What  are  your  technical  qualifications:  Specifically 

what  technical  degrees  do  you  hold? 

2.  What  is  your  technical  experience?  [What  technical  jobs 
have  you  held?] 

3.  What  is  the  mission  of  the  organization  of  which  you  are 
a  part? 

4.  What  is  your  present  job? 

5.  In  your  opinion,  what  are  the  most  important  techniques 
for  managing  technical  risk? 

6.  Based  on  your  experience,  what  is  the  most  important 
technique  for  managing  technical  risk? 

7.  Does  your  organization  have  a  specific  written  strategy 
or  policy  for  assessing  and  managing  technical  risk 
(e.g.,  document,  statement  of  policy,  operating 
instruction) ? 

8 .  Can  I  have  a  copy? 


After  each  interview  had  a  copy  of  the  preliminary  model 
for  at  least  six  working  days  the  following  questions  were 
asked : 


1.  lave  you  reviewed  and  marked  up  the  technical  risk 
management  model  to  indicate  deficiencies  or  the  model? 

2.  What  did  you  think  of  the  model? 
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Appendix  C:  Letters  Regarding  "Tin  Whisker" 

Failure  Mode 


United- #totu  £cnatc 

cawwrtu  o* 

IOVDWmUTAL  WfMU 


Mk  -r**  •  « 

Nw< 


*  lUIUtfUITTU  0* 

OVtAilAMT  COvMaWWT  MAMdtMtWT 
WAlMwOtOn.  »C  10*  II 


Pacanba r  t ,  1946 

Tha  Honorebl*  Idward  C.  Aldrldja 
Secretary 

Department  of  the  Air  Tore* 

The  Pantaion 

Vaahlni ten,  D.C.  20330-1000 
Deer  Hr.  Sscretaryi 

Xc  hu  (DM  to  ay  attention  that  lose  alaotronlc  parti 
In  weapon*  system  aiy  hewa  failed,  or  ha  Subject  to  failure# 
dui  to  tha  jrowth  of  or yitili  or  •tin  whiskers*.  Tnia 
phenooenon  appear*  to  raault  free  th«  ua«  of  pur  a  tin  In 
oloia  proilalty  to  alaotronlo  parti.  An  eiaepla  la  tha  use  of 
tin  tiaai  for  houilni  alaotronlo  hybrid#.  Wnila  several 
months  »ay  bo  required  for  tha  tin  whlakara  to  develop,  they 
eventually  grew  to  a  lan*th  auffiolent  to  brld|*  olreuiti. 

1  understand  that  at  leaat  two  najor  weapon  ayetene  require 
retrof lLilnf  sa  a  raault  of  tho  discovery  of  tin  whiskers. 
Tnii  ralid  aavaral  queatlone  X  would  Ilka  answered  by 
DactKber  19#  »946. 

Are  tin  caiaa  Tor  alaotronlc  parti  used  in  Air  Force 
weapon  syatcas.  If  ao#  do  thaaa  parti  uie  currant  of 
approa Ifltetfly  20  alcro  enters*  at  appro*  last cly  10  volte? 

Hava  an;  of  thaaa  parti  failed?  Xf  ao,  wa*  a  failure 
analysis  performed,  inoludlnf  an  analyala  to  delernir.e 
whether  tin  whlakara  ware  at  fault?  If  so,  what  wei  the 
result  of  tha  failure  analyala? 
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-c  ■ 


In  addition  to  •lictrontei  houatd  in  tin  cuu,  ara 
thara  othar  lnatanoaa  of  failed  alaetronic  part*  attributable 
to  th*  ailatanoa  of  tin  uhlaktrf?  If  ao,  pltaaa  daaorlt* 
thcaa  inataneaa. 


•  Thank  you  for  your  cooperation  In  thla  natter.  If  you 
hava  any  quaatlona  rtjardlni  thla  requaat  pltaaa  direct  your 
ataff  to  contact  Brad  Yana  of  ay  ataff  at  202-22*1-3652:. 


CLilji 


DEPARTMENT  OF  THE  AIR  FORCE 

HtAOOUAATCM  AgftONAUTlCM.  gVSTCUS  OtViSiON  {AT SCI 
WAiOnT-PArttMONAlAfCMCS  SAS*.  Or<C 

13 


i«t;  Congressional  Inquiry  Regarding  the  Crcwth  of  Crystals  or  aTln  Whiskers*  In 
electronic  Ports  (San  Carl  Covin's  1 tr ,  k  Coe  86) 

to,  HQ  USAf/LETT 

1.  Tnt  generic  phenomena,  growth  of  metallic  whiskers,  was  Identified 
during  World  War  II  with  the  growth  of  csdatua  whiskers  on  variable  plata 
capacitors  used  in  coaaunlcatton  radios.  Sines  that  tlac,  cadalua,  sine, 
or  tin  whiskers  have  been  found  In  crystal  oscillators,  capacitors,  relays, 
connectors,  tin-plated  copper  conductors  In  printed  wiring  boards,  and  tin¬ 
plated  part  leads.  Limited  occurrences  are  docuaented  in  the  Government/ 
Industry  Oats  Eachange  Prograa  (GiOCP)  data  bank  and  the  technical  litera¬ 
ture.  ml  specs  such  as  H1L-M-3B510  have  addressed  the  latter  issue  for 
several  years.  These  netsl  whiskers  are  undesirable  as  they  have  the 
potential  to  cause  electrical  disturbances  within  electronic  components/ 
system. 

2.  Ue  heve  elso  eiperlenced  soee  United  esperlencee  with  this  Issue 
within  a  few  coaponents  used  In  the  F-15  rader  and  the  aCX-65D  IR  Maverick 
Missile.  These  occurrences  cans  to  light  ee  e  result  of  ours  and  of  our 
contractor's  Internally  generated  activities.  The  specifics  era  outlined 
in  paragraph  3,  for  the  f-15  radar:  end  In  paragraph  I,  for  the  aGm-65. 

3.  Gould  Corporation  was  glean  e  contrset  by  ASD  (the  PRAM  office)  to 
demonstrate  the  effectiveness  Of  Combined  Environmental  Stress  Screening 
(CSS)  and  Destructive  Physical  Analysis  (DPI)  prograa  for  the  surveillance 
of  electronic  parte  returned  from  the  field.  The  DPA  Identified  tin 
whiskers  In  selected  alcrochlp  packages.  Subsequent  Inquires  Indicated 
that  Hughes,  Cl  Scgundo  CA,  altered  their  original  production  process 
specification  after  one  of  their  vendors  had  s  Tire  In  the  vendor's 
production  facility.  The  f-15  radar  uses  a  solder  seal  process  for 
providing  s  hermetic  seal  for  their  hybrids  using  s  pure  tin  plate  on  the 
hybrid  lids  to  facilitate  the  soldt-r  process.  Originally  these  lids  had  a 
fused  tin  coating  (healed  In  an  Oil  bath  to  2*0-260°C  to  relieve  stress 
within  the  aatcnal).  After  the  change,  the  fusing  process  was  deleted. 

The  unfused  tin  lids  have  s  high  probability  Of  growing  tin  whiskers  while  - 
the  fused  lids  are  stable.  Tin  whlskera  have  been  found  in  e  number  of 
lids  that  have  been  renoved  froa  fielded  units  or  froa  spares  Inventory.  A 
small  number  of  these  unite  have  evidence  which  Indtcetce  that  tin  whiskers 
have  shorted  between  circuit  conductors  and  part  of  the  whisker  has  vaporlied 
This  could  cause  tn  Intermittent  In  the  operation  of  the  radar,  in  the 
event  a  tin  wntsker  rests  between  two  unprotected  conductors  within  the 
hybrid,  and  If  the  hybrid  at  that  location  Is  unable  to  deliver  the  currant 
necessary  to  fuse  the  whisker,  e  hard  failure  could  result.  However,  there 
Is  no  evidence  of  whisker  generated  hard  failures  in  F-15  radar  hybrids. 
Hughes  has  modified  their  aanufacturlng  processes  to  preclude  the  growth  in 

t  in  whlskera. 


UKV  *0 

at  <m  or.  £N 


BIMTUPLA 


'UTIOS 
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*■  Th«  ACM-65  ID  H«verlck  Missile  hat  experienced  ena  problem  during  factory 
acceptance  testing  of  a  Halt  of  Acceleration  Hater  (ItCAM)  provided  by  Systron 
Penner.  TMi  device  contalna  a  moving  coll  mounted  around  a  flaed  permanent 
magnet  vhicn  aenaaa  movements  of  the  II  global  plat  fora  and  provldea  correc¬ 
tion  signals  to  the  (label  drive  motors.  failure  anaiyala  of  Inoperative 
Maks  found  during  alaalle  aaaeably  showed  tin  whisker*  to  be  growing  froa 
a  tin-plated  adder  pad  on  the  aovable  coll  which  were  abort  1  rg  the  outer 
caae.  Thia  dealdn  hea  been  modified  to  replace  the  lit-plated  adder  pad 
with  one  plated  ulth  beryllium  copper  uhtch  ha  a  eliminated  the  problem. 
Currant  production  Incorporated  this  change.  The  epprtslaately  J00  alaallea 
delivered  to  the  field  are  being  retrofitted  at  the  coctracior'a  eapenaa. 

Thia  retrofit  la  eapeeted  to  be  completed  by  the  at^uear  of  t «Pft7 .  There  la 
no  indication  this  problea  ha  a  cauaed  any  falluraa  In  sver  to  launched  by 
operational  unlta. 

5.  *i  a  result  of  thia  tiperltnce  the  following  observations  art  cade: 

a.  We  are  eble  to  respond  to  theaa  questions  because  our  disciplines 
have  been  effective  In  Identifying  and  correcting  potactlal  causes  of  field 
failure  in  electronic  equipment, 

b.  We  have  realnded  our  program  offices  of  the  potential  for  aetalllc 
whisker  growth  In  clectronie  parte  and  the  appropriate  control  procedures. 

c.  We  hava  asked  our  Air  force-laboratories  1  cl  fintlfy /develop 
suitable,  cost  effective,  alternative!  to  tin  plating  for  the  solder 
sealing  of  hcraetlc  packages. 

d.  we  recoaaend  that  the  ALCa  adopt  an  electronic  piece  part  survelllanc 
program  utilising  the  Of  A  and  CSS  process,  as  appropr'.  tie.  Prompt  identifies 
lion  of  the  causes  of  field  failures  and  the  feed  bace‘  of  this  information 

to  the  appropriate  equipment  and  part  aanufacturers  will  have  a  significant 
Upact  on  the  operational  capabilities  of  our  weapon  systems. 

6.  We  are  pleased  that  our  efforts  wera  Instrumental  in  bringing  this 
issue  to  a  successful  conclusion. 

°s.  r.  cut,)*. 

rr.rr:*-.cc  '•  f* 

TsihrcsS  f'tctcf 

Deputy  for 


1  Atch 

Sen  Levin's  ltr,  c  Dec  86 

cc:  HQ  Afi:/5D 
HQ  if-Z/rL 
AfWAL  "MLS 
AS0/Es:fa) 

ASD/TAS 

aso/x: 


167 


Appendix  D:  A  Brief  Chronology  of  the  Development 

of  the  Rocket 


AD  1232 


1903 


1914 


1915 


1920-1922 


Nov  1923 


Late  1923 


Mar  1926 


Rocket-propelled  arrows  (somewhat  like 
firecrackers)  used  in  battle  in  China 
(145:26) . 

Konstantin  Eduardoviteh  Tsiolkovsky ' s  first 
article  on  rocketry  appears  in  a  Russian 
journal;  Tsiolokovsky  conceives  of  a  rocket 
with  a  combination  of  a  liquid  oxygen  and 
liquid  hydrogen  (145:41,42). 

Robert  H.  Goddard,  American,  granted  patents 
covering  "combustion  chambers,  propellent 
feed  systems,  and  multistage  rockets" 
(145:45). 

"Robert  H.  Goddard  proved  validity  of  rocket 
propulsion  principles  in  a  vacuum,  at  Clark 
University,  Worcester,  Mass"  (42:4). 

"Goddard  developed  and  unsuccessfullv  tested 
first  liquid  propellant  engine,  using  liquid 
oxygen,  and  devised  small  high-pressure 
pumps  to  force  fuel  into  the  chamber" 

(42:16) . 

"Robert  H.  Goddard  successfully  operated  a 
liquid  oxygen  and  gasoline  rocket  motor  on  a 
testing  frame,  both  fuel  components  being 
supplied  by  pumps  installed  on  the  rocket" 
(42:17) . 

"Die  Rakete  zu  den  Pla'netenaumen  (The  Rocket 
into  Planetary  Space)  by  Hermann  Oberth  was 
published  in  Germany,  and  was  the  genesis 
for  considerable  discussion  of  rocket 
propulsion"  (42:18). 

"Robert  H.  Goddard  launched  the  world's 
first  liquid-fueled  rocket  at  Auburn, 

Mass.,  which  traveled  184  feet  in  2  1/2 
( two-and-a-hal f ]  seconds.  This  event  was 
the  "Kitty  Hawk"  of  rocketry"  (42:21). 
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Nov  1929 

Dec  1930 

Apr  1932 

Mar  1935 

Mar  1936 

Oct  1936 


Dec  1941 

Jun  1942 

Oct  1942 


Famed  aviator  Colonel  Charles  A.  Lindbergh 
visited  Goddard.  Through  Lindbergh’s  effort 
Goddard  was  awarded  a  $50,000  grant  from  the 
Guggenheim  Fund  and  another  grant  from  the 
Carnegie  Institution  (145:46). 

"Robert  H.  Goddard  fired  11-foot  liquid  fuel 
rocket  to  a  height  of  2,000  feet  and  a  speed 
of  near  500  mph  over  Roswell,  New  Mexico" 
(42:26) . 

"First  flight  of  Goddard  rocket  with 
gyroscopically  controlled  vanes  for 
automatically  controlled  flight"  (42:29). 

"Goddard  rocket  attained  altitude  of  7,500 
feet  in  New  Mexico"  (42:33). 

"Robert  H.  Goddard's  classic  report  on 
"Liquid  Propellant  Rocket  Development" 
reviewing  his  liquid-fuel  rocket  research 
and  flight  testing  since  1919,  was  published 
by  the  Smithsonian  Institution"  (42:34). 

"Lt.  John  Sessums  (AAC)  visited  Robert  H. 
Goddard  to  officially  assess  military  value 
of  Goddard's  work.  He  reported  that  there 
was  little  military  value,  but  that  rockets 
would  appear  useful  to  drive  turbines" 
(42:43) . 

First  test  of  the  German  V-l  rocket  at 
Peenmunade  (145:105). 

"First  test  of  tne  German  A-4  (V-2)  rocket 
unsuccessful  at  Peenerinde,  Germany" 

(42:43) . 

■'First  successful  launch  .d  flight  of  5  1/2 
( f ive-and-a-hal f ]  ton  German  A-4  rocket  (V- 
2)  at  Peenemunde,  which  traveled  120  miles" 
(42:44) . 
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Appendix  E: 

1905 

Dec  1938 

Jan  1939 

Aug  1939 

1939 

1940 

1941 

Dec  1941 
May  1942 

1942 


Chronology  of  Some  Major  Events  Regarding 
Atomic  Bomb  Development 


Albert  Einstein  publishes  the  Special  Theory 
of  Relativity  that  states  the  equivalence 
between  energy  and  mass  (46:829). 

Enrico  Fermi  receives  Nobel  Prize  for 
experiments  involving  low  velocity  neutron 
bombardment  of  elements  (45:324). 

Refugee  Austrian  scieatists  Lise  Meitner  and 
Otto  Frisch  explain  (via  Niels  Bohr)  that 
German  scientists  discovered  "low-velocity 
neutrons  cause  the  uranium  nucleus  to 
fission,  (break  apart]".  .  .  much  energy  was 
released  in  the  process"  (46:324);  numerous 
laboratory  experiments  begun  (46:324). 

Albert  Einstein's  letter  (dated  2  August 
1939)  addressed  to  President  Roosevelt, 
warns  that  the  Germans  may  be  developing  the 
first  atomic  bomb,  and  that  recent  work  by 
L.  Szilard  and  Enrico  Fermi  has  established 
technical  feasibility  (41) . 

Alfred  A.  0.  Nier  achieves  the  first 
separation  of  the  uranium  isotopes  U-235  and 
U-238  at  the  Universi  .y  of  Minnesota 
(47:517) . 

First  funds  for  atomic  research  given  to 
United  States  scientists  (46:831). 

"The  Columbia  chain-reaction  experiment  with 
natural  uranium  and  carbon  yielded  negative 
results"  (45: 325) . 

The  U.S.  enters  World  War  II  (45:325). 

"The  decision  is  made  to  proceed  on  all 
promising  production  methods  [for  obtaining 
fissionable  materials]"  (45:325). 

Army  Corps  of  Engineers  opens  an  office  in 
New  York  City  called  the  Manhattan  Engineer 
District  office  (45:325). 
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Dec  1942 

Jun  1942 

July  1942 

Apr  1943 

1943 

Jul  1943 

Sep  1944 

Apr  1945 

Jul  1945 

6  Aug  1945 
9  Aug  1945 
16  Aug  1945 
Aug  1946 
Apr  1956 


"First  nuclear  chain  reaction  successfully 
accomplished  at  the  University  of  Chicago" 
(42:44). 

J.  Robert  Oppenheimer  appointed  "director  of 
Project  V,  the  group  that  was  to  design  the 
actual  weapon"  (45:325). 

Experimental  data  reveals  "plutonium  does 
give  off  neutrons  in  fission,  more  than 
Uranium-235"  (45:325). 

Seth  Neddermeyer  proposes  an  "implosion" 
method  of  attaining  supercritical  mass 
(45:325) . 

John  von  Neumann  and  Edward  Teller  support 
the  implosion  approach  (45:325). 

Summer/Fall  "Gun"  Method  of  attaining  super 
critical  mass  of  1943  being  pursued 
(45:325) . 

"It  (becomes]  clear  that  the  plutonium  gun 
could  not  be  built"  (45:325),  therefore,  the 
implosion  approach  is  pursued. 

"First  reactor  at  Hanford,  Washington, 
turned  on  but  .  .  .  promotly  turned  itself 
off"  (45:325). 

Harry  Truman  briefed  on  the  Manhattan 
Project,  after  just  two  weeks  as  president 
(45:325)  . 

"First  test  atomic  device  exploded  in  New 
Mexico"  (42:50). 

An  atomic  bomb  (using  gun  approach)  is 
dropped  on  Hiroshima,  Japan  (42:51;  47:830). 

An  atomic  bomb  (using  implosion  method)  is 
dropped  on  Nagasaki,  Japan  (42:51;  47:830). 

"World  War  IT  ended  with  Japanese  surrender" 
(42 : 51) . 

The  Soviet  Union  tests  their  first  atomic 
device  (47:831) . 

"Dr.  John  von  Neumann  was  awarded  the  Enrico 
Fermi  Award  for  anticipating  the  importance 
of  the  high  speed  computer  in  nuclear 
development  programs.  .  .  ."  (42:82). 
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Appendix  F:  A  Brief  Chronology  of  the  Development 

of  the  Laser 


1917 

1954 

1958 

1959 

1960 

Jul  1960 

1961 

1961 

1961 

1963 


The  principles  of  spontaneous  and  stimulated 
emission  of  radiation  recognized  by  Albert 
Einstein  (44:686;  101:23). 

A  working  maser  device  is  built  by  Townes, 
James  P.  Gordon  and  H.  J.  Zeiger  using 
ammonia  gas;  subsequent  improved  models  are 
built  by  Bell  Laboratories.  MASER  is  the 
acronym  for  Microwave  Amplification  by 
Stimulated  Emission  of  Radiation  (135:234). 

Arthur  L.  Schawlow  and  Charles  H.  Townes 
proposed  a  technique  to  obtain  an  optical 
MASER.  Such  a  device  would  be  based  on 
Light  Amplification  by  Stimulated  Emission 
of  Radiation  (LASER)  (134:234). 

A  helium-neon  gas  laser  is  proposed  by  Ali 
Javan  of  Bell  Telephone  Laboratories 
(135:230). 

A  successful  prototype  helium-neon  gas  laser 
built  and  demonstrated  by  Ali  Javan,  W.  R. 
Bennett,  Jr.,  and  D.  R.  Harriott  (135:230). 

The  first  successful  construction  and 
operation  of  a  laser  achieved  by  T.  H. 

Maiman  of  the  Hughes  Aircraft  Company  using 
a  synthetic  ruby  crystal  (134:234;  135:227). 

The  first  gas  laser  is  operated  (Bell 
Laboratories)  (135:266). 

Ivan  P.  Kaminow  of  Bell  Laboratories 
demonstrates  a  high-frequency  laser 
modulator  that  uses  a  potassium  dihydrogen 
phosphate  (KDP)  crystal  (117:333). 

Other  lasers  are  constructed  using:  (1) 
different  crystals,  and  (2)  "a  mixture  of 
hydrogen  and  helium  gas"  (23:495). 

First  liquid  laser  successfully  operated  by 
General  Telephone  and  electronics  (101:259). 
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Appendix  G:  A  Brief  Chronology  for  Achievement  of  Man_  on 

the  Moon 


Sept  1955 

4  Oct  1957 

31  Jan  1958 

2  Apr  1958 
12  Apr  1961 

5  May  1961 

25  Jan  1962 


President  Eisenhower  assigns  "highest 
national  priority"  (42:79)  to  the  research 
and  development  of  the  Intercontinental 
Ballistic  Missile  (ICBM) . 

The  Soviets  launch  Sputnik  I--the  world's 
first  man-made  satellite  —  into  Earth  orbit 
(11:32;  42:91;  145:  164)  . 

United  States  launches  Explorer  I  satellite 
into  Earth  orbit  (11:32;  42:91;  145:164). 

Eisenhower  creates  the  National  Aeronautics 
and  Space  Agency  (NASA)  (11:32;  145:165). 

Soviet  Cosmonaut  Yuri  Gagarin  "orbited  earth 
in  a  five-ton  capsule  becoming  the  first  man 
in  space"  (1  32 ) . 

United  States  President  John  F.  Kennedy,  in 
a  speech  to  the  Congress,  announces  the  goal 
of  a  manned  rocket  mission  the  moon 
(116:736;  145:170). 

Congress  approves  a  development  program  for 
the  three-stage  Saturn  V  launch  vehicle; 
program  given  the  highest  priority 
(  14  5: 1 7  0 ) . 


1962 


1964 


1965 


9  No/  1966 


U.s.  Ranger  3  probe  missed  the  moon 
entirely.  Ranger  4  lands  on  moon  but  due  to 
malfunction  sends  back  no  data.  Ranger  5 
misses  moon  and  loses  power  due  to  solar 
cell  malfunction  (145:191). 

U.S.  Ranger  6  probe  launched;  unsuccessful 
due  to  burned-out  electrical  system.  Ranger 
7  appropriately  redesigned  and  successfully 
transmitted  pictures  of  the  moon  (145:19). 

Soviet  space  probes  Luna  5,  7,  and  8  crash 
onto  the  moon.  Luna  6  misses  the  moon 
entirely  (145:191). 

First  launch  of  a  Saturn  V  vehicle 
successful  (90:60;  145:171). 
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3  Feb  1966 

30  May  1966 

Apr  1968 

27  Jan  1967 

Oct  1968 

Dec  1968 

Mar  1969 


Soviet  space  probe  Luna  9  achieves  first 
soft  landing  on  the  Moon's  surface 
(145: 191-192) . 

U.S.  Surveyor  probe  launched;  makes 
successful  soft  landing  on  moon  and 
transmits  pictures  (145:193). 

Second  launch  of  a  Saturn  V  vehicle — part  of 
the  unmanned  Apollo  6 — is  unsuccessful  due 
to  failures  of  three  J-2  engines  on  the 
vehicle  (90:60;  145:171). 

Apollo  1.  Three  astronauts  killed  by  fire 
during  full  launch  rehearsal  (91:53). 

Apollo  7.  First  manned  flight  of  Apollo 
Command  and  Service  vehicles  conducted  in 
Earth  orbit;  test  fired  Service  Module's 
engine  by  firing  it  (while  in  orbit) 
automatically  as  well  as  manually;  heat 
shield  performance  checked  during  reentry 
(91:53) . 

Apollo  8.  Man's  first  flight  around  the 
moon;  147-hour  mission.  Twenty  hours  spent 
circling  the  moon.  Photos  of  potential 
landing  areas  taken  (91:53). 

Apollo  9.  First  manned  flight  with  the 
Lunar  Module;  ten-day  flight  confined  to 
earth  orbit;  separation,  rendezvous  and 
docki.ig  practiced  between  the  Lunar  Module 
(LM)  and  the  Command  Module  (CM)  (91:53). 


May  1969  Apollo  10.  "Successful  dress  rehearsal  for 
the  actual  moon  landing  [which  occurred]  two 
months  later  .  .  .  first  time  complete 
Apollo  spacecraft  had  orbited  the  moon" 
(91:53).  This  was  an  8-day  mission  (91:53). 

Jul  1969  Apollo  11.  First  man  on  the  moon;  crew  of 
Neil  Armstrong,  Edwin  Aldrin,  Mike  Collins 
(91:54) . 
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Appendix  H:  A  Brief  Chronology  of  the  Development 

of  the  Jet  Aircraft 


Nov  1917 

Dec  1919 

Mar  1922 

Jun  1920 

Sep  1928 

During  1929 

During  1931 

During  1932 

Apr  1937 


"Committee  on  Light  Alloys  established 
within  NACA  to  develop  new  metals  for 
aeronautical  use,  constructor  Jerome  C. 
Hunsaker  was  Navy  member"  (42:7). 

"The  Aeronautical  Engineering  Society  was 
organized  at  MIT"  (42:11). 

"NASA  report  No.  159  on  Jet  Propulsion  for 
Airplanes,  by  Edgar  Buckingham  of  the  Bureau 
of  Standards,  pointed  out  that  jet  fuel 
consumtion  would  be  four  times  that  of 
propeller  engine  at  250  mph,  but  that 
efficiency  of  jet  increased  at  higher 
speeds"  (42:14,15). 

"Initial  flight  of  all-metal  airplane 
(Gallaudet)  designed  by  Engineering  Division 
at  Wright  Field"  (42:17). 

"Frank  Whittle,  RAF  officer  and  engineer, 
obtained  British  patents  for  turbojet 
engine"  (42:27) . 

"NACA  annual  report  indicated  that 
aerodynamic  efficiency  may  be  increased  by 
applying  the  principle  of  boundary  layer 
control  to  the  wings  and  possibly  other 
parts  of  the  airplane"  (42:26). 

"Bureau  of  Standards  made  a  number  of 
experiments  to  determine  whether  the  thrust 
reaction  of  a  jet  could  be  increased,  and 
tested  combinations  of  jets"  (42:28). 

"German  engineer,  Paul  Schmidt,  working  from 
design  of  Lorin  tube,  developed  and  patented 
a  ramjet  engine  later  modified  and  used  in 
the  B-l  Flying  Bomb"  (42:29,30). 

"Frank  Whittle's  first  gas  turbine  engine, 
the  U-type,  was  static  tested"  (42:35). 
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1940 


Mar  1941 


Nov  1941 


Jul  1942 


Jul  1942 


Oct  1942 


Jul  1943 


Aug  1943 


"Committee  of  the  National  Academy  of 
Sciences  reported  that  operation  of  turbine 
wheels  at  temperatures  up  to  1,500  [degrees] 
F  might  soon  be  possible  because  of  U.S.  and 
foreign  development  of  high-temperature 
alloys"  (42:40) . 

"NACA  established  Special  Committee  on  Jet 
Propulsion  to  review  early  British  reports 
on  the  Whittle  engine,  which  subsequently 
aided  development  of  TG-100  turboprop  engine 
by  GE  and  the  19-B  turbojet  by  Westinghouse . 
Dr.  W.  F.  Durand  was  called  out  of 
retirement  to  head  this  committee"  (42:41). 

"Italian  jet-propelled  Caproni-Campini 
airplane  flown  475  kilometers  in  2  hours  11 
minutes  from  Turin  to  Rome,  by  Mario  de 
Bernardini"  (42:42). 

"German  ME-262  turbojet  fighter  flown  on 
spectacular  flight  test,  concluding  a  series 
begun  in  May"  (42:44). 

"First  U . S . -designed  jet  engine  successfully 
demonstrated  at  Langley  Laboratory,  the  NACA 
Jeep,  which  was  never  flown  but  proved 
invaluable  for  continued  NACA  research  on 
gas-turbine  jet  propulsion"  (42:44). 

"First  U.S.  jet-propelled  aircraft  flight, 
by  an  Airacomet  Bell  XP-59A  (powered  by  two 
1-16  engines  developed  by  General  Electric 
from  the  British  Whittle  prototype) ,  made  at 
Muroc  Dry  Lake,  CA,  with  Robert  Stanley  as 
pilot"  (42:44). 

"First  turbojet  engine  completed  for  the 
Navy,  the  Westinghouse  19A,  completed  its 
100-hour  endurance  test"  (42:45). 

"German  turbojet  fighter,  a  Messerschmitt 
ME-262,  demonstrated  before  Adolph  Hitler  in 
East  Prussia"  (42:45). 
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Appendix  I:  Letter  Regarding  Electrostatic  Dischara 


to 

*rt-  ot  £h  (  PA  ) 


DEPARTMENT  OF  THE  AIR  FORCE 


Kl 


7  AUG  1965 


Product  Assurance  Policy,  Electrostatic  Discharge  (ESD) 


TO>  ASD/AE  ASD/AF  ASD/01  ASD/KW  ASD/TA 

ASO/TP  ASO/TS  ASD/TW  ASD/ IT  ASD/TZ 

((or  Assistant  for  Product  Assurance) 

1.  Electrostatic  discharge  hat  become  well  recognized  as  a  source  o f  failure, 
delay  and  cost  In  electronics  esseably  as  well  ss  of  Incipient  failures  affect¬ 
ing  systea  reliability  and  durability  tn  operational  service.  Parts  such  as 
alcroc  ircul  ts ,  discrete  semiconductors,  thick  and  min  file  resistors,  hybrids 
and  chips  are  known  to  be  Sensitive  to  this  phenomenon.  In  conjunction  with  our 
general  workmanship  and  quality  program  requirements ,  we  have  come  to  vspect 
mat  our  contractors  dealing  with  such  devices  will  have  adopted  effective  ESD 
protection  practices. 

2.  In  a  recent  special  survey  conducted  by  AFCMD.  they  not  only  found 
deficiencies  In  contractor  ESD  protection  efforta,  but  also  eipcrlenced  a  lack 
or  contractual  leverage  on  aome  programs  with  which  to  secure  contractor 
Improvements  in  the  ESD  area.  DOD-STD-1606  (1900),  "Electrostatic  Discharge 
Control  Program  for  Protection  of  Electronic  Parts,  Assemblies  and  Equipment," 
is  the  preferred  contractual  vehicle  for  specifying  the  need  for  ESD  controls 
and  should  be  cited  In  tne  product  specification  for  all  systems  containing  ESD 
sensitive  devices. 

3.  In  a  review  conducted  for  Can  McMullen,  we  found  that  DOD-STO-1686  Is 
generally  cited  as  a  requirement  on  the  newer  ASD  programs j  but  since  product 
spec l f leal ions  on  several  current  programs  predate  the  existence  of  this 
standard,  it  has  not  always  been  made  contractually  applicable.  In  such  cases, 
w«  advise  looking  into  the  adequacy  of  your  contractor's  ESD  control  practices 
In  conjunction  with  the  APPRO  to  be  sure  that  there  are  no  alsund«r stano 1 ngs  of 
our  contractual  Intent.  It  la  our  opinion  mat  the  MIE-Q-98S8A  quality  program 
specification  should  suffice  as  the  governing  requirement  In  such  Instances. 

a.  Our  near  term  policy  Is  that  DOD-5TD-16S6  be  referenced  In  all  new  product 
specifications  for  aystams  containing  atailc^tenstt  lve  devices.  This  standard 
will  t.e  added  to  the  listing  of  workmanship  standards  on  the  ASD  Fora  *5. 

Quality  ‘.saursnee  Memorandum.  In  the  Aocg  term,  the  Avionics  Integrity  Program 
nlL-PhlME  standard,  now  under  development,  will  encompass  the  DOC-STD-I660 
provisions  and  will  be  the  primary  means  for  contr«ctually  invoking  ESD  and 
other  integrity-related  requirements  In  avionics  systems  scqulsttlon. 

5.  Should  you  have  any  questions  tbout  this  policy  or  Its  implementation, 
please  contact  the  undersigned  or  Mr  George  Thltlcn,  ASD/EhSl,  eatcnslon  HMO, 


Assistant  for  Product  Assurance 
Deputy  for  Engineering 


cc:  ASD/EhSl 
EKAS 
PnOO 
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Appendix  J :  Model  Review  Comments  from  WRDC 


PHASE 


TECHNICAL  CONTINGENCY  PLANNING 


ACC I DENTS /HAZARDS  IN 
ANALOGOUS  OR  PREDECESSOR 
PROGRAMS 


( 


IDENTIFY  NATIONAL  SUPER- 
EXPERTS  AND  FACILITIES  WITH 
EXPERIENCE  CORRESPONDING 
TO  EACH  "UNKNOWN  UNKNOWN" 


IDENTIF 


_  _  ___125RIN  TABLE  _ 

j  (SKANj/i-r ROWE: 4  0) 

&r/j/r  'D2>  4 *;*/'*& 

/Jfjy-iyjJirs.  T>d 

wr-  w/a  j  r  ‘Identify  and  pr i ok  it  i  le 


^  -  POTENTIAL  HAZARDS 


TECKK 

( ;:oo : 


rLAN  : CR 

:CAL  CONTINGENCIES 
:80;THUR: 17) 


PURSUE  AND  DEMONSTRATE 
A  LOW-TECKNOLOGY 
BACKUP/FALLSACK  APPROACH 
“^5/ FOR  EACH  CRITICAL  SYSTEM 
X  COMPONENT  OR  MODULE. 
(GOOD : 26  2 ) 

*<<°  3Z0QU1/O  /) 


4*<hcu  or  P 

as  CeO.  sc*"i/£JPAJ  P***u£L 
'D€U£(.c0»£vf  oP*  /?  topped  ^£,J  ‘r 
ef/JK  a/0  3'0  iiV/- 
ft  JO  fig  M  ft t/£  fffiTTZA/alJ  A/-V" 

CfiJ£  ftoAL  Anoj  - 

dp>PP&?  iivp  id<ir  /.OJ  Z.'/'/i  cjoJaso  y  ' 

'^U^rry  J£*,<U4  -  M  ’Af^c/P  Co  J 


‘ZOPsSeJ  ?/  a/££o(T/#£ 

’Z'Vi/cJ'J  £4.  Ccjr  {jA6*r<xa/»*Cf)  /V 
JJ//0A7O  3'rp  ■T'Jr?  '3JU'€7~ 


OJ^S  v/^v  ^P.'S4  &c/r  P/PA 
P^PAA/C/A^T  P//.r/A  77A:£'?<a  35  jJ. 


^V>-  Cfr*r*ofQ 


u- 
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I 


ARE  THE  PEP.S0I.1IZL 
EXPERIENCED  WITH  ' 

PRIMARY  TECHNICAL  APPROACr. 

SELECTED  ? 

OmmCM'-*  sj 

J_FtfOK  M«  OAriAi0 
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!  ! 


i 


I  i  !r 

illiT 


co::0UC7  FE AS ISILITY/CONCEPT 

STL'OIES 

(TR'-'R:  Is) 

i 


OBTAIN  RELEVANT 
PREVIOUS  RESEARCH 

(AIIOE:  3  .  E-10,  IS) 

AVOID  PITFALL  CF 
TECHNICAL  OVEP.-OPTIK- 
ISH  REGARDING  THE  EASE 
‘of  applying  new  op.  MODIFY¬ 
ING  ESTABLISHED  TECHNOLOvY 

i rox : 1 2 e :  cAoit:=i 
DAF1:  20 

AVOID  pitfall  OF  PROCEEDIN': 
WITH  NEW  TECHNOLOGY  WHEN 
r>:lSTIKO  OR  MODIFIED  OFF- 
THE-SHELF  ECUIPKENT  WILL 
SATISFY  REQUIREMENTS . 

(  > 

£H*oU>  >->°r 

CJ  &Ztu,-rn  Qr>T£<rr,6~ 


V£u  07 
TECHNICAL  EXPERTISE 
REQUIRED  VS  AVAILABLE 
(ROWE:  <0;  THUP.:  !«■) 


i  i 

i 

i 


i  : 


CE?  ICISNCIES/LIM. 
'OF  EXISTING  TEST 
TIES  AND  EQ'-’IPME 
(PEN'T:  lO 


DESIGN  OR 
ITEMS  NCT 


(F.KOI 


:  o 


SAL'l 


) 
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Appendix  K:  Interview  with  Dr.  Keith  Richev. 
Technical  Director  of  WRDC.  28  July  1989 


Interviewer:  Sir,  what  are  your  technical  qualifications? 

Dr.  Richev:  The  highest  degree  is  a  PhD  in  Aerospace 
Engineering  from  the  University  of  Michigan. 

Interviewer :  What  is  your  technical  experience? 

Dr.  Richev:  I'll  give  you  a  biographical  sketch,  then 
I'll  fill  in  some  details.  I  have  28  years  of  experience 
in  the  government  laboratory  systems.  In  fact,  all  in 
Flight  Dynamics  laboratories  of  AFWAL  now  [reorganized 
into]  WRDC.  And  for  the  last  2  years  in  WRDC  front  office. 
So,  all  of  my  experience  except  for  2  years  I've  gone  to 
school  full-time  with  a  PhD  level  with  the  flight  dynamics 
laboratory,  and  with  WRDC. 

Interviewer :  What  is  the  mission  of  the  organization  of 

which  you  are  a  part? 

Dr.  Richev:  Basically  research  and  development. 

Technology  development  for  the  Air  Force.  So,  its  a 
focused  application  of  all  technologies.  Aerospace, 
electrical — any  technologies  that  we  deal  with,  focused 
toward  Air  Force  applications.  The  basic  research  through 
advanced  development. 

Interviewer:  What  is  your  present  job? 

Dr.  Richev:  Technical  Director  of.  WRDC,  (Chief  Scientist 
of  the  Chief  Scientists,  if  you  will). 

Interviewer :  In  your  opinion,  what  are  the  most  important 

techniques  for  managing  technical  risk? 

Dr.  Richev:  Well,  first  of  all  I  think  you  have  to 
understand  the  problem.  And  understand  as  much  as  you  can 
about  the  technical  characteristics  of  a  problem.  Try  to 
be  as  analytical  as  you  can.  I  guess  the  engineer  in  me  is 
coming  out.  You  try  to  be  as  analytical  as  you  can  but  to 
recognize  a  lot  of  things  that  you  can't  be  analytical 
about.  But  I  think  you  need  to  sit  back  and  try  to 
understand  the  problem  as  much  as  you  can.  Then  try  to 
understand  the  various  facets  of  the  problem.  I'll  try  to 
break  it  down  into  some  elements.  Try  to  understand  how 
the  element's  interact.  And  then  try  to  get  a  feel  for  the 
degree  of  r isk--whether  its  achieving  an  objective,  or 
schedule  or  cost,  or  anything  else.  A  degree  of  risk  that 


might  be  associated  with  each  element  and  the  interaction 
among  the  key  elements,  and  then  try  to  construct  a 
critical  path  function  that  shows  you  where  the  highest 
risk  elements  are  and  their  importance  to  the  outcome  of 
the  system.  Something  might  be  very  high  risk  but  not  very 
important.  So  you  wouldn't  care  about  managing  it  at  all. 
Let  it  be  high  risk.  So  what.  It  doesn't  affect  the 
outcome.  Something  else  could  be  of  lower  risk  but 
extremely  critical.  So,  that  a  little  bit  of  risk  in  a 
critical  element  could  have  a  large  affect  on  the  outcome 
of  the  system.  Try  to  identify  those  elements  or  most 
likely  combination  of  elements  because  things  don't  usually 
happen  singularly.  They  usually  happen  in  multiples. 

Those  combination  of  elements  which  are  the  highest  risk  to 
achieving  the  desired  outcome.  This  assumes  that  you  have 
a  pretty  good  idea  of  what  the  outcome  is,  which  is  not  a 
good  assumption  in  all  cases.  But,  let's  say  you  have  a 
good  idea  of  the  desired  outcome;  so  having  gone  through 
this  analysis  it  could  be  qualitative  or  quantitative.  But 
you  have  some  idea  of  the  elements  and  the  degrees  of  risk, 
and  the  importance.  Now  we  have  identified  the  elements  or 
combination  of  elements  that  have  the  largest  impact  on  the 
desired  outcome.  And  as  I  said  before  some  of  those 
elements  might  be  high  risk  factor;  others  might  be  the  low 
risk  factors.  But  the  criteria  is  the  effect  on  the 
outcome.  So,  having  identified  the  elements  of  the  problem 
that  have  the  largest  effect  cn  the  outcome,  now  we  would 
be  in  the  position  where  we'd  try  to  manage  the  risk  of  the 
elements.  To  me  managing  the  risk  means  to  identify  what 
the  risk  is  at  the  beginning  of  the  process,  then  taking 
positive  action  to  reduce  the  risk  to  an  acceptable  level 
at  the  conclusion  of  the  process — recognizing  that  you 
can't  wait  until  everything  is  at  zero  risk  to  complete. 
Most  R  and  D  [Research  and  Development]  projects  will  be 
completed  without  having  risk  equal  to  zero  in  all  factors. 
There  is  going  to  be  some  risk  at  the  end.  But  it  has  been 
managed  to  be  by  the  process.  It  has  been  managed  to  be 
reduced  by  the  many  factors  to  an  acceptable  level.  Now  a 
case  in  point  in  the  Air  Force  labs  WRDC  particular,  is 
reducing  risk  to  an  acceptable  level  so  that  the  next  stage 
of  technology  development  can  begin  or  follow  on.  Which 
would  be  engineer  development.  Category  6.4  budget,  which 
is  not  done  by  the  laboratory  but  is  done  by  the  system 
program  offices.  Whoever  is  applying  the  technology  would 
take  our  product.  Our  product  is  technology  developed  to 
such  a  point  that  there  is  enough  confidence  in  a  low 
enough  risk,  but  I  wouldn't  say  zero  risk.  Enough 
confidence  and  low  enough  risk,  so  that  they  can  apply  the 
technology  with  confidence  in  the  next  step  of  development. 
Which  is  6.4  engineering  development.  And  tnat  is  the  way 
it  is  in  the  Air  Force.  That  is  the  way  it  is  in  the 
evolution  of  the  technology.  Now  my  perception  would  be 
that  in  any  given  project  there  will  still  be  risk,  when  we 
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hand  it  off  from  6.3  to  6.4.  The  risk  will  be  further 
reduced  and  managed  in  the  6.4  and  beyond.  But  the 
laboratories  wouldn't  have  much  to  do  with  that--with  the 
risk  management  6.4  and  beyond,  only  on  consultation 
basis.  Because  our  job  is  to  develop  the  technology  to  a 
point  and  then  hand  off.  We  very  seldom  stay  with  a 
technology  all  the  way  through  to  its  final  application. 
It's  just  the  way  the  Air  Force  is  organized.  We  have 
laboratories,  SPOs,  users.  We  have  Logistics  Command  and 
so  forth  and,  so  on.  If  we  took  the  attitude  in  the 
laboratories,  that  we  have  to  reduce  risk  to  an  absolute 
zero  level  before  we  transition  it,  number  one:  we'll  keep 
the  technologies  too  long  and  they  wouldn't  transition  to 
the  Air  Force  as  fast  as  they  should.  Two,  it  would  be 
prohibitively  expensive  because  the  last  10  percent  of  a 
project,  in  terms  of  a  full  risk  reduction,  is  very 
expensive.  And  you  want  to  only  do  that  in  given 
applications.  Because  if  we  would  say  that  we  had  to 
reduce  the  risk  to  zero  for  any  and  all  applications  since 
we  don't  know  what  the  applications  will  be,  we'll  have  to 
reduce  it  for  any  possible  application,  and  that  would  take 
many  more  resources,  in  terms  of  dollars  and  people,  than 
we  have  available.  So,  we  make  a  conscious  decision  as  to 
when  to  turn  it  loose.  To  use  a  reasoned  judgment  on  what 
that  hand  off  is.  We  sometimes  learn  after  we  hand  it  off 
that  we  didn't  do  enough.  There  is  a  major  surprise 
somewhere  in  the  application  of  our  technology.  And  if 
were  real  honest  with  ourselves,  we'd  say  we  should  have 
caught  that.  We  should  have  known  that  would  happen.  The 
users  of  our  technologies,  in  that  case,  give  us  a  poor 
mark.  The  user  comes  back  and  says:  you  guys  gave  me 
foreign  technology,  and  we  say:  well  we  probably  did.  We 
usually  make  an  excuse  that  we  didn't  have  enough  money  or 
didn't  have  enough  time.  And  there  is  always  the  unknown 
unknowns.  Once  in  awhile  a  technology  goes  sour  in  a  big 
way  after  it  leaves  the  laboratory.  So  we  try  to  minimize 
that  by  lessons  learned  and  by  understanding  the 
application  of  our  technology  to  the  field  after  they  leave 
the  laboratories.  We  try  to  learn  as  we  go,  and  minimize 
our  cases  but,  they  can  still  happen. 

Interviewer :  Based  upon  your  experience,  what  is  the  most 

important  technique  for  managing  technical  risk? 

Dr.  Richey:  It's  hardly  a  technique,  but  I  would  say  the 
answer  to  your  question  would  be  knowledge;  as  much 
knowledge  as  you  can  possibly  have  on  the  physical  and  non¬ 
physical  phenomenon  that  you're  dealing  with.  We  don't 
have  perfect  Knowledge  of  any  perimeters,  but  I  would  say 
that  the  single  most  important  to  know  to  manage  risk  is  to 
understand  the  physical  and  nonphysical  driver. 
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Interviewer:  Does  your  organization  have  a  specific 

written  strategy  or  policy  for  assessing  and  managing 
technical  risk? 

Dr.  Richev:  Not  specifically.  There  are  no  documents  that 
you  can  pick  out — that  I  am  aware  of  at  least  for  the 
center  level — that's  called  risk  management  or  even  the 
risk  assessment.  On  the  other  hand  it  goes  through 
everything  we  do.  You  know  we  manage  risk  every  day.  It's 
part  and  parcel  to  the  management  of  technology.  And 
knowledgeable  people,  scientist  and  engineers,  and 
managers,  and  project  managers,  and  technical  development 
are — whether  they  recognize  are  managing  risk  every  day. 

But  to  my  knowledge  there's  not  a  written  policy.  Now 
there  have  been  some  studies  done  on  risk  assessment  by  the 
laboratories;  risk  assessment  and  perhaps  risk  management. 
And  the  most  knowledgeable  guy  I  would  know  of  that  has 
done  this  in  WRDC  is  Dr.  Squire  Brown  [Interviewed]. 
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Appendix  L:  Interview  with  Dr.  Harris  Burte. 
Chief  Scientist  of  WRDC  Materials  Lab. 

17  August  1989 


Dr.  Burte:  I  understand  better  what  you  are  driving  at  as 
we  look  at  this  outline  (Preliminary  Risk  Management 
Model] — it's  a  rather  thorough  and  complete  outline  of  many 
things.  I  have  two  comments.  One  has  to  do  with  the  idea 
that  risk  reduction  means  different  things  depending  upon 
whether  you're  in  the  process  of  developing  a  system  to 
meet  a  stated  operational  deficiency — at  one  extreme  or  at 
the  other  extreme  exploring  or  doing  what  you  can  to  foster 
the  transition  into  use  of  new  technological  possibilities. 
Those  are  quite  different  things.  In  the  middle  between 
those  (extremes]  you  might  have  the  question  of  pursuing  a 
national  goal  that's  less  well  defined  than  a  required 
operational  deficiency  that  you  have  to  pursue  or  pursuing 
the  possibilities  of  meeting  some  national  vision  or  goal 
that  somebody  such  as  the  President  enunciates.  Examples 
of  these  might  be  "put  a  man  on  the  moon"  or  the  SDI 
project  v/here  the  President  or  some  other  decision  making 
authority  has  a  vision  of  something  and  says  "I  want  to  get 
there."  Those  are  three  different  levels.  In  one  case  you 
have  a  required  operational  need  and  [you  want  to  solve] 
"What's  the  best  way  to  meet  it  at  this  time  ?"  That's 
akin  to  the  systems  development  process.  In  the  other  case 
you  have  a  vision  of  something  in  the  future  and  you  are 
exploring  possibilities.  In  the  third  case  you're 
developing  the  capability  to  meet  these,  possibilities.  In 
the  final  case,  you  have  a  technological  possibility  which 
might  be  applied  to  a  wide  variety  of  operational  needs,  or 
perhaps  only  a  small  number  of  operational  needs  and  you 
are  exploring  what  you  have  to  do  to  transfer  laboratory 
concepts  or  demonstrations  into  something  that  can  actually 
be  used.  I  have  the  feeling  that  you  treat  all  of  these  a 
bit  the  same  in  this  [model]  and  I  think  that  you  might 
want  to  explore  the  extent  to  which  they  are  different. 

When  we're  working  against  a  potential  threat  or  a 
statement  of  operational  deficiency,  to  use  your  words  [in 
the  model],  the  ways  of  doing  it  tend  to  fall  into  many  of 
the  things  that  you're  talking  about  here  [in  the  model]. 
That's  what  a  SPO  does  when  it  tries  to  organize  of 
analyze:  What  are  the  best  ways  of  meeting  this?  What  are 

the  cheapest  ways?  What  v/ill  give  me  the  most  bang  for  the 
buck?  How  do  I  cope  with  the  risk?  In  these  ways,  balance 
the  risks  against  the  payoff  and  make  sure  that  I'm  not 
taking  undue  risks  for  the  time  and  the  budget  I  have  to  do 
my  system.  In  the  "mandated  national  priority"  sort  of 
thing,  such  as  "put  a  man  on  put  a  man  on  the  moon  or  do  an 
SDI,  there  has  to  be  some  guidance  as  to  timeline.  At 
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different  stages  you  can  take  a  lot  of  risk,  you  can 
explore  a  lot  of  possibilities,  or,  in  something  as  simple 
as  SDI — "let's  provide  a  defensive  shield  for  the  country" 
--risk  isn't  much  applicable  there  in  the  initial  stages. 
The  question  is  to  explore  what  the  possibilities  are  more 
at  that  stage.  And  then  when  you  have  the  possibilities, 
to  assess  the  question  of  what  it  takes  to  get  them  into 
use.  Risk  becomes  a  factor  as  a  function  of  time  and 
budget  available  to  put  it  into  use.  In  the  exploration 
stage,  risk  is  considered  differently.  Risk  is:  Which 
things  are  most  likely  to  be  promising  in  the  long  range? 
That's  different  than:  How  much  time  and  money  will  it 
take  me  to  get  all  the  bugs  out  of  an  actual  operational 
system?  Those  are  quite  different  questions.  Finally,  in 
the  case  of  taking  something  where  some  degree  of 
laboratory  feasibility  has  been  demonstrated — I  talked 
about  this  to  some  extent  last  time  [earlier  interview ] -- 
and  transitioning  it.  The  question  is  now:  you've  got  a 
laboratory  thing,  but  if  somebody  is  building  a  system,  he 
has  a  certain  time  and  budget  and  he's  got  to  meet  that 
time  and  budget  so  he  has  to  know  that  the  things  he 
chooses  for  his  system  are  compatible  with  his  kind  of 
budget.  An  industrial  analogue  to  that  is:  if  somebody 
wants  to  bring  a  new  product  to  the  market,  he  does  a 
return  on  investment  calculation.  But  to  do  the  return  on 
investment  calculation  in  the  commercial  world  often 
requires  a  great  deal  of  information  about  the  new  product, 
and  he  requires  the  knowledge  that  he  will  be  able  to  do 
these  things  [that]  strange  things  won't  pop  up  and  hit  him 
in  the  face  and  destroy  him. 

Therefore,  the  decision  to  go  ahead  with  a  new 
product — the  capitalizing,  build  the  factory,  do  all  the 
advertising  and  everything  to  get  a  new  product  to  the 
market--is  ~  big  multi-buck  decision.  If  there  is  a  lot  of 
risk  in  terms  of  the  new  product--we ' re  not  really  going  to 
be  able  to  do  it  or  strange  things  are  going  to  happen,  or 
it's  going  to  take  us  a  lot  more  money  or  a  lot  more  time 
to  bring  it  to  the  market  place--then  the  decision  takers 
are  likely  not  to  go  with  the  new  product.  We  may  go  with 
it  foolishly  as  Rolls  Royce  did  [mentioned  earlier 
interview]  and  go  bankrupt  first.  So,  those  are  different 
elements  of  risk  reduction.  In  some  cases  it  is  a  case  of 
risk  assessment.  In  the  case  of  a  system  developer,  it's 
more  a  case  of  risk  assessment.  You  don't  have  that  much 
time  to  reduce  risk  once  you  decide  that  you  want  to  go  to 
a  system.  In  the  case  of  pushing  new  technology  forward, 
then  there  is  the  question  of  learning  what  the  strange 
things  are  or:  How  can  I  scale  it  up?  Or  making  sure 
there  are  no  gaps  in  it  or  learning  more  about  the  behavior 
of  the  new  product  so  that  you  can  really  assess  what  the 
new  benefits  will  be  versus  the  cost  of  it.  In  the  case  of 
exploring  new  possibilities,  like  in  the  SDI  example, 
that's  a  totally  different  question.  There  one  has  to 
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[ask],  which  possibilities  are  most  likely  to  give  me  the 
biggest  payoff?  Now  at  a  much  later  stage,  somebody  might 
decide  now  I  want  to  set  up  an  SDI  system.  Of  the 
different  possibilities  available,  in  which  one  is  the  risk 
adequate  for  my  purpose?  That  becomes  a  risk  assessment 
issue . 

Interviewer :  So,  risk  assessment  doesn't  start  until  you 

completely  or  comprehensively  explore  the  possibilities? 

Dr.  Burte:  Well,  you  can  always  do  risk  assessment  at  any 
time.  People  in  the  military  in  particular--you ' re 
spending  money  to  take  chances.  But  at  any  given  time  you 
may  say  I  have  got  to  use  this  technology  [or]  I  won't  be 
able  to  do  what  I  want  to  do.  But  you  can  take  an 
assessment  of  what  the  risk  is  and  ask  yourself  then.:  Are 
you  willing  to  do  that?  You  shouldn't  just  go  ahead  without 
thinking  about  it.  So  those  are  the  main  points  as  I  see 
it.  And  I  think  those  are  a  bit  different  and  I  think  you 
have  a  bit  of  all  of  those  confused.  In  other  words  for 
the  three  things  of  mandated  major  possibility  like  SDI  or 
the  Orient  Express  [National  Aero-Space  Plane]  at  the 
present — that's  for  the  question  of  the  early  stage.  For 
each  of  them  [the  three  sources  of  risk  shown  in  the  model] 
the  question  of  risk  is  considered  differently.  If  it  is  a 
requirement,  I  want  to  do  this  in  a  finite  time — like  put  a 
man  on  the  moon.  That  can  be  like  a  system  requirement  and 
you  can  say,  well,  for  that  time  and  budget  I  have,  is  the 
risk  tolerable  or  can  I  overcome  the  thing?  If  it  is  a 
mandated  national  thing  that  says:  I  will  build  an  Orient 
Express  that's  a  different  thing.  There  we're  exploring 
the  possibilities — or  SDI  as  initially  put  forth.  And  if 
in  fact  you're  restricting  yourself  to  where  the  President 
says  I  want  a  definitive  capability  at  such  a  time  at  such 
a  budget,  then  it  is  a  risk  assessment  issue.  But  there 
may  be  time  in  that  to  do  the  development  of  new 
technology.  Much  of  what  I  was  talking  about  wasn't  in  the 
risk  assessment  aspect  it  had  to  .  with  the  question  of 
converting  new  technology  to  a  poA  where  somebody  could 
do  a  credible  risk  assessment.  If  it's  too  early,  you're 
making  too  many  assumptions  about  the  risk;  you  don't  have 
the  knowledge  of  what  the  risks  are.  Part  of  risk 
reduction  is  just  getting  the  information  so  that  you  can 
make  a  risk  assessment.  Okay,  those  are  the  main  things 
that  I  want  to  say  and  that  may  or  may  not  apply  very  much 
to  this  [risk  management  model].  If  you're  sticking  very 
much  to  [the  case  where]  the  technology  is  essentially 
there  to  do  the  job  or  to  do  something  like  the  job  that  I 
want,  within  the  time  and  budget  that  I  have,  trying  to 
pick  the  technologies  that  will  best  work  for  the  purpose, 
what  you've  written  tends  to  apply  considerably.  If  you're 
getting  into  other  things  like  an  SDI  system,  or  finding 
uses  for  new  technology  such  as  VHSIC — that's  much  of  what 
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I  was  talking  about  beforehand  [in  the  earlier  interview] 
and  you  way  not  be  getting  into  that.  You  may  want  to  look 
over  what  you're  saying  from  the  viewpoint  of  it  [the 
model]  creeping  into  that.  Because  that's  risk  reduction. 
Here  you're  talking  about  risk  assessment  and  assuring  that 
the  risk  in  the  first  case  where  you've  got  a  definitive 
set  of  requirements  and  now  you  want  to  make  sure  that  the 
approaches  that  you  take  fit  that.  That's  not  really  risk 
reduction.  That's  risk  assessment  and  risk  selection — 
unless  you  have  some  time  to  explore  different 
possibilities.  If  you  have  time  to  explore  different 
possibilities,  that  becomes  something  like  risk  reduction. 
But  many  of  these  [projects]  don't  give  you  much  time  to 
explore  different  possibilities.  .  .  .  Risk  management 

gets  to  be  not  so  much  risk  reduction  as  risk  assessment — 
choosing  the  right  things — when  you're  in  the  systems 
world.  You  should  have  a  lot  of  that  done  before  you  go 
into  the  system  selection.  Essentially,  risk  management  is 
different  than  concept  exploration. 

The  other  thing  I  want  to  talk  about  is  what  is  the 
role  of  a  government  lab  in  this.  There  I  feel  strongly 
that  a  government  lab  such  as  our  lab  or  any  other 
government  lab — one  of  their  highest  callings  is  to  play  a 
major  leadership  role  in  doing  any  of  the  things  I  have 
talked  about.  A  government  lab  hopefully  is  wedded  only  to 
the  Air  Force--in  the  case  of  an  Air  Force  Lob.  It's  a 
neutral  ground;  it  has  no  ax  to  grind.  Hopefully  it  has 
within  itself  excellence  which  is  the  peer  of  anybody  in 
the  country,  not  just  bureaucratic  smart  buyers.  Really 
competent  people  who  know  the  technology  and  are  on  the 
forefront  of  leading  the  development  of  new  technology. 
People  such  as  this  should  be  deeply  involved  in  any  of  the 
things  I've  talked  about:  risk  assessment  for  giving  a 
clear-minded  unbiased  viewpoint  of  what  we  know  and  what  we 
don't  know.  And  an  assessment  of  how  serious  is  what  we 
don't  know  and  how  long  it  will  take  us  to  fill  in  the  gaps 
and  is  that  commensurate  with  the  budqet  and  time  available 
for  the  project  that  you've  set  up.  In  the  case  of 
exploring  a  spectrum  of  new  possibilities,  the  excellence 
involved  here  is  that  which  can  exercise  leadership  in  the 
nation  as  a  whole.  To  have  very  good  people  explore  new 
possibilities  in  other  than  a  wish-is-the-fathar-to-the 
conclusion  mode.  Exploring  them  enthusiastically  but 
always  asking:  What  do  we  know  and  what  don't  we  know  and 
what  might  the  pitfalls  be?  The  government  lab  can  again 
exercise  tremendous  leadership  here  and  act  as  a  governor 
on  that  process  because  the  people  who  are  advocates  for  a 
given  activity  tend  to  enthusiastically  look  at  the 
potential  for  success.  As  [the  late  Nobel  pt'ize  winning 
physicist  Richard  P.  ]  Feynman  said — I  think  his  words  were 
something  like  in  the  Challenger  [Space  Shuttle  disaster 
review]  committee — I  think  his  words  were  something  like 
"you  can't  fool  Mother  Nature."  That's  the  point.  People 
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who  really  understand  the  science  and  the  technology  and, 
by  being  wedded  to  the  Air  Force,  make  sure  we  ask  the 
right  questions.  The  other  point  which  is  the  business  of 
assessing  and  understanding  what  it  takes  to  reduce  the 
risk.  That’s  all  the  work  that  takes  place  as  you  go 
between  the  typical  6.2  work  and  the  introduction  of 
something  into  a  system.  It  is  that  work — when  I  talked  to 
you  last--that  I  was  saying  is  deficient  in  many  areas  of 
the  country. 
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Appendix  M:  Interview  with  Dr.  Jim  Olsen,  Chief  Scientist 
of  WRDC  Flight  Dynamics  Laboratory  and  Dr.  Arnold  Maver. 
Assistant  for  Research  and  Technology.  25  July  1989 


Interviewer :  Sirs,  what  are  your  technical  qualifications? 

Dr.  Olsen:  I  hold  a  bachelors,  masters,  and  PhD  all  in 

Aerospace  Engineering. 

Dr.  Haver:  I  have  a  bachelors  in  Mechanical  Engineering,  a 
masters  in  Engineering  Mechanics,  and  a  PhD  in  Mechanical 
Eng ineer ing . 

Interviewer :  What  is  your  technical  experience? 

Dr . _ Olsen:  I've  done, aircraft  performance,  instability  and 

control  and  then  went  into  structures,  vibrations,  flutter 
unsteady  aerodynamics. 

Dr .  Maver :  I  was  a  mechanical  engineer  with  Bell  Telephone 
laboratory  supporting  the  mechanical  and  thermal 
development  of  electronic  devices,  electron  tubes,  and 
integrated  circuitry,  and  also  principal  engineer.  I  was 
also  Cryogenic  Engineer  with  Grumman  Aerospace,  also  I  was 
the  principal  engineer  with  Cornell  Aeronautical  labs,  SCAD 
Missile  project.  with  the  Air  Force  worked  on  R&D  into 
aircraft  environmental  control  systems,  Electronics 
coolant,  a  little  bit  of  windshield  impact  research.  I 
have  to  get  into  a  lot  of  diversified  things  because  of  the 
position  I  hold  as  Chief  Engineer  of  Mechanical  Division 
where  we  cover  landing  gear,  escape  systems  and 
environmental  control,  and  ballistics  research. 

Interviewer :  What  is  the  mission  of  the  organization  of 

which  you  are  a  part? 

Dr.  Olsen:  Our  mission  is  to  do  the  research  and  the 
development  that  gives  the  Air  Force  the  capability  to 
develop  a  better  airplane  and  to  improve  the  ones  they  have 
or  to  give  them  a  range  of  options  of  what  they  might  want 
to  choose  for  the  next  aircraft.  We  do  a  research  and 
development  and  aero  dynamics,  structure,  flight  control, 
system  and  all  the  subsystems. 

Interviewer:  What  are  each  of  your  present  jobs? 

Dr.  Olsen:  I'm  the  Chief  Scientist  of  the  (Flight 
Dynamics)  the  laboratory. 


194 


Dr.  Maver:  For  the  ihicle  Subsystem  Division,  I'm  the 
division's  Chief  Assistant  for  Research  and  Technology. 

Interviewer:  In  each  of  your  opinions,  what  are  the  most 

important  techniques  for  managing  technical  risk.? 

Dr.  Olsen:  Risk  is  something  I  don't  think  we  manage  that 
much.  We  generally  working  on  trying  to  prove  sorething  is 
feasible  or  something  can  be  done.  How  about  assessment? 

I  guess  explicitly  I  don't  think  we  practically  speaking 
really  evaluate  the  risk.  We  talk  about  risk.  But  we 
generally  determine  if  the  idea  is  possible.  In  a  very 
general  way  if  it's  practical  At  least  I  don't  know  of 
any  way  really  quantify  risk  in  deciding  to  go  with  higher 
risk  or  lower  risk. 

Dr.  Maver:  We  compare  various  alternative  concepts  and  do 
payoff  analyst.  We  try  to  establish  the  feasibility  and 
the  process  of  establishing  feasibility.  I  think  we 
discover  that  some  things  are  not  feasible  therefore  too 
risky  and  other  things  are  more  feasible.  And  those  that 
survrve  these  various  levels  of  investigation,  they 
eventually  get  developed.  And  our  contractors  evaluated 
alternative  approaches.  They  propose  approaches  and  if  we 
think  they're  credible  by  virtue  of  having  checked  the 
concepts  versus  the  physical  laws  of  nature  then  we  just 
keep  going  as  long  as  it  remains  feasible.  If  we  run 
against  an  obstacle.  Because  we've  in  the  business  of 
eliminating  risk  or  sorting  out  risky  things  from 
impossible  things,  just  by  continuing  to  develop  them.  If 
we  hit  a  stone  wall  then  the  project  stops,  or  some  other 
alternative  attack  is  pursued. 

Dr.  Olsen:  I  guess  if  you're  developing  an  airplane  you 
want  to  look  at  several  different  approaches  of  doing  a 
particular  part  of  the  mission  or  building  part  of  the 
airplane.  You  look  at  the  risk  there.  But  I  guess  in  my 
experience  in  the  61XX  and  62XX  areas  really  risk  is  not 
something  you  evaluate  very  much  Maybe  in  63XX  development 
there  might  be  some  efforts  they  pursue  or  don't  pursue 
because  of  risks. 

Interviewer:  What  is  the  most  important  technique  for 
managing  technical  risk? 

Dr.  Olsen:  I'll  go  back  and  say  that  I  don't  think  that  we 
really  have  some  kind  of  graduated  scale  of  risk  that  we 
manage  against.  We  try  to  put  our  resources  where  they  pay 
off  the  most  and  we  try  to  do  things  that  we  can  prove  are 
feasible  and  will  have  high  payoff. 

Interviewer:  How  do  you  improve  that  feasibility? 
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Dr.  Olsen:  Well,  I  guess  you  come  up  with  the  possible 
design  or  application  or  implementation  you  can  have.  But, 
again  the  feasibility  is  sort  of  either  it  is  or  it  isn't. 
It's  like,  it  works  or  it  doesn't  work.  For  instance  we 
developed  something  called  a  active  control  system  to 
aircraft  flutter  the  vibrations  of  the  wings.  And  we 
demonstrated  to  our  satisfaction  that  the  61XX  and  62XX 
level  that  we  could  make  it  work.  We  went  into  the  wind 
pile  and  built  it.  Yes,  this  thing  works.  And  we  know  how 
to  do  it.  When  the  question  about  risk  comes  that  sort  of 
moves  up  into  another  ball  game.  That  question  is:  can  we 
get  the  F-16  program  office  to  use  this  method  to  improve 
their  airplane?  At  this  point  they're  saying  either  it  is 
to  expensive,  or  it  doesn't  come  at  the  right  time  or  is  a 
too  high  of  a  risk.  But  again  its  a"go"  or  "no  go" 
decision.  So  I  don't  know  what  the  best  method  is  of 
managing  risk. 

Interviewer :  Does  your  organization  have  a  specific 

written  strategy  or  policy  for  assessing  and  managing 
technical  risk? 

Dr.  Olsen:  Hot  that  I'm  aware  of.  We  have  one  set  of 
programs  that  we  pursue  called  the  laboratory  independent 
programs  where  the  director  tells  us  that  he  wants  us  to  go 
in  to  very  risky  areas.  Because  of  the  perception  that 
sometimes  we  get  too  conservative.  So  there  is  a  whole  pot 
of  funds  set  aside  to  go  into  those  high  risk  areas  to  see 
what  you  can  do. 

Dr.  Maver:  We  make  use  of  analysis  to  see  if  our  concept 
is  feasible.  Using  mathematical,  physical  models  of  things 
to  the  extent  we  can  anticipate  that.  Another  step  is  to 
actually  do  development  testing  to  do  tests  to  determine 
some  phenomenon  performs.  Or  we  can  get  the  performance  out 
of  the  phenomenon  that  we  look  for  in  magnitude.  And  we 
eventually  try  to  develop  a  demonstration.  And  there  is  a 
lot  of  iterative  steps  between  analysis  and  testing  until 
it's  clear  the  right  combination  of  the  attributes  and 
performance  characteristics  have  been  achieved.  We  also 
worry  about  cost,  weight.  Is  the  thing  to  heavy?  is  it 
palatable?  It  can't  weigh  to  much,  it  can't  cost  to  much, 
and  it  has  to  be  energy  efficient.  So,  it  is  a  balancing 
act,  and  sometime  we  switch  concept  in  order  to  alleviate 
some  of  the  objectionable  characteristics. 

Dr.  Olsen:  I  think  that  you're  going  to  find  risk  in  the 
laboratories  is  not  the  same  as  risk  in  the  SPO.  He’s  got 
a  plane  to  build  and  a  time  to  build  it  in,  and  a  cost  to 
build  it  in.  He's  definitely  looking  for  the  lowest  risk 
way  of  getting  done  what  he  has  to  get  done. 
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Dr.  Haver:  We  though  provide  him  with  the  information  on 
which  concepts  are  feasible,  because  they  have  been  tried 
in  the  laboratory  previously.  I  think  a  SPO  director  would 
be  foolish  not  to  require  that  something  have  been 
demonstrated  previously  in  an  advanced  development  program 
where  we  spend  a  lot  of  money  just  trying  to  do  something 
for  the  first  time  to  prove  that  it  can  be  done.  And  its 
done  in  a  quasi  generic  fashion  so,  that  when  the  SPOs  want 
to  build  a  particular  aircraft,  they  derive  some  confidence 
measure  from  our  having  done  a  generic  version  of  that 
previously. 

Dr.  Olsen:  We  demonstrate  whether  it  v;orks  at  all,  or 
whether  if  it  can  be  done  at  all. 

Interviewer :  Can  I  have  a  copy  of  any  strategy  for 

minimizing  technical  risk? 

Dr.  Olser.:  I  don't  know  of  any. 

Dr.  Mayer;  Well,  I  suppose  that  you  could  consider  the  way 
we  categorize  the  levels  of  research  as  basic  research 
where  natural  phenomenon  are  investigated  basically  to 
understand,  the  exploratory  research,  where  we  try  to  apply 
these  phenomena  and  configure  them  to  work  and  perform  some 
useful  function  on  an  aircraft  or  Air  Force  contracts. 

Then  there  is  advance  development,  where  some  form  of 
realistic  full  scale  demonstrations  of  the  subsystem  or 
product  is  nducted.  And  then  there  is  the  64XX,  the 
engineering  evelopment ,  for  a  particular  weapon  system. 
That  is  what  ASD  and  the  SPOs  is  exposed  to. 

These  also  correspond  to  budget  categories.  The 
"appropriate  money  set  aside  are  kind  of  inversely  with  the 
risk.  The  further  stages  really  get  more  expensive, 
because  you  have  to  pay  attention  to  more  realistic 
details.  You  have  to  make  it  look  like  an  airplane,  make 
it  look  like  a  wing,  a  landing  gear,  or  a  tire.  Whereas  at 
the  early  stages,  we  can  just  play  with  the  materials, 
rubber  or  what  ever. 

Dr.  Olsen:  I  think  that  is  a  good  point,  because  I  guess 
when  you  move  up  to  basic  research,  exploratory.  Are  you 
familiar  with  the  61XX,  62XX,  63XX?  The  61XX  is  basic 
research,  62XX  is  exploratory  development.  And  as  you  move 
into  the  other  categories  like  63XX  and  64XX  then  risk.  .  . 

In  the  63XX  program  they  do  look  for  different  options 
to  achieving  the  goal  and  evaluating  the  risk.  [Is  6lxx, 

6 2 XX,  63XX ,  part  of  the  labs?j  Yes,  even  some  64XX. 

Dr.  Mayer:  We  actually  build  some  windshields  through 
retrofits.  But  we  primarily  use  61XX,  62XX,  63XX. 
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Dr.  Olsen:  The  programs  are  61XX  for  basic,  62XX  for 
exploratory,  63XX  for  advanced,  and  64XX  is  for  engineering 
development.  I  guess  there  is  otner  [budget  categories] 
categories  65XX  but  I  don't  know  what  they  are.  Maybe  you 
should  talk  to  some  63XX  people.  I  was  trying  to  think  of 
the  guys  who  does  the  63XX  programs  for  structures. 

There's  a  guy  you  should  talk  to  named  Vern  Johnson. 
Johnson's  phone  number  is  X55664.  And  I  know  that  they 
look  at  part  of  their  63XX  program,  and  they  look  at 
several  different  approaches  to  get  a  particular 
improvement  in  an  airplane.  And  one  of  the  things  they 
evaluate  is  the  risk.  Risk  versus  cost  performance.  They 
try  to  avoid  risk  because  they're  now  in  the  exploratory 
and  basic.  We're  more  "blue  sky"  and  willing  to  be  more 
daring,  and  try  different  things.  As  it  gets  closer  to 
something  that  goes  on  an  airplane  people  begin  to  manage 
the  risk  as  you  say  by  eliminating  risky  things  and 
pursuing  less  risky  design  options. 

Dr.  Olsen:  1  think  we  just  didn't  really  get  into  it  [risk 
reduction]  until  we  start  talking  about  the  63XX  that 
Arnold  [Dr.  Mayer]  brought  up. 

Dr.  Mayer:  That's  where  the  aircraft  or  aerospace  aircraft 
manufacturers  get  into  it.  They  are  usually  the  performers 
of  those  contracts.  And  they  force  them  to  do  things  the 
way  they  already  know  how  to  do  them.  To  some  extent  that 
minimizes  risk  too. 

Dr.  Olsen:  I  don't  know  if  you  would  want  to  minimize  the 
risk.  I  think  its  just  an  all  out  effort.  Maybe  later  on 
when  you  got  to  the  point  of  where,  you've  got  a  bomb  built 
and  you're  going  to  put  it  on  an  airplane  to  drop  it,  then 
minimizing  risk  might  become  an  issue. 

Dr.  Mayer:  Well,  they  had  to  work  out  a  lot  of  problems. 
How  do  you  make  sufficient  quantities  of  these  radioactive 
materials  for  the  bomb.  And  I  guess  they  have  to  develop 
some  of  the  factories.  Some  of  the  uranium  purifications 
centrifuge  and  whatever  is  involved. 

Dr.  Olsen:  And  then  there  were  a  couple  of  different 
process.  I  remember  that  General  Groves  was  forcing  the 
people  to  pursue  at  least  two  different  manufacturing 
methods.  The  manufacturing  methods  and  minimizing  risks. 
And  giving  himself  at  least  an  option. 

Dr.  Mayer:  Now  we  use  that  too,  we  call  it  fly  offs.  We 
usually  try  to  have  two  different  teams  go  and  build 
something.  Then  they  fly  off  the  best  result  so  you'll  at 
least  have  the  tetter  of  two  approaches. 
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Dr.  Olsen:  Of  course  the  engineers  wanted  to  keep  the 
parallel  path  going  cs  long  as  possible  [for  the  atom 
oomb] .  Eventually  General  Grove  forced  them  into  a 
decision  to  go  one  way  or  the  other.  Whether  the  way  they 
do  the  enclose  or  how  they  manufacture  the  stuff.  That's  a 
good  history  to  look  at.  I  always  thought  that  he  was  just 
the  civil  engineer  that  built  the  things  and  put  them 
together.  The  last  book  I  read,  he  really  played  a  very 
strong  management  role.  He  wasn't  just  someone  that  was 
standing  on  the  side  lines  waiting  on  for  results,  but  he 
was  really  driving  those  guys. 

Dr.  Maver:  Well,  I  don't  know  whether  putting  the  man  on 
the  moon  is  one  of  your  cases  [it  is].  I  recall  that  one 
way  they  minimize  risk  is  that  they  had,  one  engineer  on 
every  piece  of  hardware.  There  was  one  engineer  that  did 
nothing  but  one  particular  valve  for  three  years.  And  he 
was  100  percent  responsible  for  that  valve.  The  design  and 
performance.  I  guess  if  you  put  a  lot  of  people  on 
something  and  dedicate  them  you  can  reduce  risk. 
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Interviewer:  Sir,  what  are  your  technical  qual i f icat ic; . 

Dr .  Riles:  I  hold  a  bachelor  in  Mechanical  Engineering, 
bachelors  in  Aeronautical  Engineering,  and  a  roasters  and 
PhD  in  Electrical  Engineering. 

Interviewer:  vJhat  is  your  technical  experience? 

Dr.  Riles:  At  one  tiroe  I  was  the  program  manager  for  NASP 
(National  Aero-Space  Plane)  National  back  in  the  60s'  when 
it  was  called  the  Aero-Space  Plane,  now  it  is  the  National 
Aero-Space  Plane.  I  have  worked  in  research  and 
development.  I  have  worked  in  the  acquisition  business  in 
Aeronautical  Systems  Division.  When  I  was  in  the  service  I 
was  a  Maintenance  and  Communication  Officer.  So  I  kind  of 
had  the  full  spectrum  of  creating  the  idea  to  hew  you  keep 
it  working  on  the  flight  line. 

Interviewer :  What  is  the  mission  of  the  organization  of 

which  you  are  a  part? 

Dr.  Riles:  Research  and  development,  and  Aerospace 
Avionics . 

Interviewer :  What  is  your  present  job? 

Dr.  Riles:  Chief  Scientist  of  the  Avionics  Laboratory, 
Wright  Research  and  Development  Center. 

Interv iewer :  In  your  opinion,  what  are  the  most  important 
techniques  for  managing  technical  risk? 

Dr.  Riles:  So  you're  saying  that  the  first  elements  of 
feasibility  have  been  assessed? 

Interviewer :  Yes. 

Dr.  Riles:  Well,  I  think  probably  the  most  used  technique 
here  at  ASD  (Aeronautical  System  Division)  is  to  put  people 
on  it  with  systems  engineering  experience,  that  have 
developed  similar  kind  of  things  in  the  past.  To  try  as 
much  as  possible  to  support  at  least  a  couple  of  competing 
approaches.  These  days  with  the  funding  we  have  its  a 
little  difficult  to  fund  competing  approaches.  So 
sometimes  you  prematurely  have  to  decide  on,  if  you  will, 
"one  horse  to  ride."  And  after  you  have  gone  a  year  or  two 
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downstream,  you  find  you're  on  the  high  risk  horse.  If  you 
could,  you  could  fund  the  beginning  approaches.  The  other 
is  to  solicit  the  opinion  of  many  people  in  the  other 
services  and  other  national  agencies,  to  help  you  in 
establishing  their  course  of  engineering  development  of 
whatever  you're  talking  about,  and  we  do  a  lot  of  that.  We 
bring  in  expert  co,  .ittees,  scientific  advisory  boards.  We 
bring  in  expert  consultants  from  industry  and  from  other 
sources  other  national  labs  to  help  us. 

Interviewer:  Based  on  your  experience,  what  is  the  most 

important  technique  for  managing  technical  risk? 

Dr.  Riles:  I  think  that  probably  the  one  that  seems  to 
have  worked  most  often  is  make  sure  you  use  people  that 
have  experience.  What's,  the  old  saying?  "When  you  don't 
know  what  direction  you're  going,  almost  any  direction  will 
do"  . 


Interviewer:  Does  your  organization  have  a  specific 

written  strategy  or  policy  for  assessing  and  managing 
technical  risk? 

Dr.  Riles:  Well,  see  the  way  you  justified  the  risk, 
you've  almost  taken  the  problem  out  of  the  laboratory. 
Because  what  we  do  is  come  up  with  some  new  ideas  and  we 
carry  them  through  to  some  kind  of  feasibility 
demonstration.  Then  we  try  to  encourage  users.  "Users"  is 
a  rather  broad  term  to  us  it  can  be  people  in  the  operating 
commands.  It  could  be  people  in  the  intelligence  agencies, 
it  could  be  the  engineering  people  in  ASD  who  support  SPO's 
in  going  out  and  acquiring  weapon  systems.  So,  once  you 
take  it  out  of  that  "demonstrate  the  feasibility"  it's  a 
kind  with  technical  viability  of  a  concept  for  use  of  the 
Air  Force  or  other  services.  Its  then  something  that's 
outside  the  laboratory  to  manage  to  get  into  a  weapon 
system. 

Interviewer:  Let's  say  that  you  have  a  lot  of  feasible 

options,  how  does  the  lab  decide  which  one  has  the  least 
technical  risk? 

Dr.  Riles:  Well,  you're  kind  of  getting  into  the 
philosophy  of  how  do  we  prioritize  the  work  we  do.  What 
you're  talking  about  is  an  investment  strategy.  And  what 
we  do  is  that  in  the  various  disciplines  that  we're 
responsible  for  like  electronic  warfare,  offensive 
avionics,  developing  things  to  go  out  and  find  targets, 
identify  them  and  provide  the  weapons  capability  to  strike 
them.  We  build  programs.  First  of  all,  we  provide  some 
top  down  guidance,  and  how  much  money  we  have,  what  does 
front  office  think  we  should  be  in  what  kind  of  businesses, 
which  ones  look  like  we  should  invest  in  the  heavier,  and 


201 


then  from  the  bottom  they  build  the  program.  They  put 
together  the  various  ideal  that  we  want  to  pursue.  Now 
some  of  them  are  way  out.  Some  of  them  are  closer  in. 

Then  we  have  what  we  call  Kind  of  a  rack.  It's  a  period  of 
time,  usually  the  early  part  of  the  given  year  where  we 
develop  the  next  year  program.  And  during  that  time  we 
have  to  evaluate  the  maturity  of  the  concept,  and  how  much 
risk  there  is  involved  in  doing  this  and  is  it  absolutely 
frivolous  to  go  out  and  do  something  or  is  there  some 
possibility  it's  going  to  pay  off.  Usually  we  take  expert 
opinion  inside,  in  doing  our  first  ranking  of  these  ideal. 
If  we  have  questions  about  the  liability  of  some 
particular  piece  of  technology,  we  have  experts  that  we  can 
go  to:  scientific  advisory  boards,  defense  science  boards. 
We  all  have  people  in  the  industry  we  can  talk  to  or 
technical  compatriots  in  universities  that  we  can  go  access 
and  get  their  opinions  to  help  us  decide  whether  or  not  we 
have  ranked  or  prioritized  or  agree  to  fund  this  idea  in 
the  appropriate  fashion.  Or  is  there  to  much  ri.sk  in  it? 
It's  just  not  worth  investing  in  now.  So  I  would  have  to 
say  that  expert  opinion  is  probably  the  most  used. 

Interviewer:  Is  there  any  policy  within  the  organization 

for  managing  technical  risk? 

Dr.  Riles:  a  policy  document? 

Interviewer:  Yes. 


Dr.  Riles:  No . 

Dr.  Riles:  One  thing  that  I  didn't  mention  is  that  we  have 
a  voting  process,  which  involves  risk  in  it  and  its 
essentially  a  Delphi  method  of  choosing  things  to  fund.  We 
typically  take  a  vote  after  we  make  sure  that  everybody 
understands  the  particular  things  that  we  want  to  fund. 

And  by  the  way  you  talked  about  it.  We  have  about  700 
different  things  going  on  at  once,  and  about  100  new  items 
are  added  each  year  and  about  a  100  die.  So  we  typically 
have  a  steady  state  of  about  700  programs,  about  100  new 
ones  starting,  and  100  old  ones  completing.  So,  what  we  do 
is  rank  the  continuing  ones  with  the  new  ideas,  and  we  use 
Delphi  to  do  that.  And  a  part  of  Delphi  involves 
consideration  and  risk.  We  will  fund  things  that  have  high 
risk  if  it  has  high  pay  off. 
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Appendix  O:  Interview  with  Dr.  Squire  Brown.  Chief 
of  the  Technology  Assessment  Division.  Wright 
Research  Development  Center.  28  July  1989 


Interviewer :  Sir,  what  are  your  technical  qualifications? 

Dr.  Brown:  I  have  a  bachelor  of  science  and  a  PhD  in 
Aerospace  Engineering,  some  22  years  of  professional 
experience  mostly  related  to  aircraft  technology,  aircraft 
design,  development  of  aircraft  related  technologies. 

Interviewer:  What  is  your  technical  experience? 

Dr.  Brown:  I  primarily  worked  in  aerodynamics  and  flight 
vehicle  design  and  analysis,  from  aerodynamics  research  to 
configuration  development,  analysis  of  operational 
effectiveness . 

Interviewer:  What  is  the  mission  of  the  organization  of 

which  you  are  a  part? 

Dr.  Brown:  The  Technology  Assessment  responsibility  is  to 
examine  and  create  systems  concepts  for  future  mission 
requirements  for  the  Air  Force,  and  to  examine  the 
technologies  that  are  essential  for  the  development  or 
employment  of  that  system.  What  are  the  technologies  that 
are  needed  to  be  effective  in  the  future  scenarios? 

Interviewer :  What  is  your  present  job? 

Dr.  Brown:  I'm  chief  of  the  Technology  Assessment  Division 
[Wright  Research  and  Development  Center  (WRDC) ] . 

Interviewer :  In  your  opinion,  what  are  the  most  important 

techniques  for  managing  technical  risk? 

Dr.  Brown:  Deal  a  little  bit  with  the  term  risk.  Can  we? 
Would  you  like  to  help  me  out  with  what  you  mean  by  risk? 

Interviewer:  Assuming  that  technical  feasibility  has  been 

established,  how  do  we  go  from  the  establishment  of 
feasibility  to  minimizing  the  risk  of  actually  achieving 
that  project.  I  guess  a  contemporary  example  would  be  the 
Space  Defense  Initiative  or  the  contemplated  X-ray  laser.  . 


Dr.  Brown:  the  accepted  approach,  and  I  believe  one  that 
works  well,  is  one  that  would  lead  through  a  series  of 
hardware  demonstration  programs.  It  might  be  peculiar  to 
the  technologies,  but  it  might  be  a  combination  of 


203 


component  bench  testing,  simulation  flight  demonstration, 
either  on  a  test  bed  or  perhaps  as  part  of  a  new  prototype 
system.  An  X-vehicle  [Experimental  vehicle]  or  some 
suitable  modification  of  a  flight  vehicle.  But  ultimately 
risk  in  the  sense  that  we're  talking  about  here  almost  need 
verification  of  the  article.  That  would  go  into  a  deployed 
system. 

Interviewer:  Based  on  your  experience,  what  is  the  most 

important  technique  for  managing  risk? 

Dr .  Brown:  It's  a  loaded  question.  Let  me  try  to  break  it 
apart  a  little  bit.  We  approach  the  demonstration  in  terms 
of  getting  the  risk  out  as  an  accepted  way  of  doing 
business.  We  recognize  that  technologies  come  along  and 
show  promise  in  the  feasibility  demonstrations,  and  we  know 
that  they're  going  to  be  critical  to  some  capacity  in  the 
future  system.  And  you've  got  to  simply  understand  that 
you  got  to  go  ahead  and  do  the  flight  demonstration  part, 
or  a  suitable  demonstration.  Much  of  our  energy  is 
consumed  ,  I  wouldn't  say  so  much  as  with  management  as 
with  the  advocacy  and  developing  the  programs  to  carry  out 
this  demonstration.  The  management  would  only  occur  in  so 
far  as  our  management  of  programs  themselves.  We  are  very 
much  captive  of  the  institutional  way  that  we  do  business 
of  the  POM  [Program  Objective  Memorandum]  cycle.  Are  you 
familiar  with  the  61XX,  62XX,  63XX  programs?  [Yes]. 

As  we  come  out  of  61XX  is  a  basic  research,  much  of 
the  in-house  work;  much  of  the  feasibility  work  is  the  62XX 
arena  limited  amount  of  dollars.  As  we  come  out  of  62XX 
saying  this  is  the  technology  that  we  really  need  to  flight 
demonstrate  as  for  an  example.  Much  of  our  energy  is 
devoted  to  getting  the  dollars  to  get  the  program  put  in 
place  to  do  that  flight  demonstration.  A  very  excellent 
example  at  the  moment  is  the  2-D  [two-dimensional] 
vectoring  nozzles  that  are  flying  on  the  F-15.  Would  you 
like  me  to  go  through  that  example  for?  [I  think  that  I 
would  prefer  you  to  go  through  it.] 

It  is  probably  as  good  an  example  as  we  have  at  the 
moment.  It's  relevant  to  your  questions  about  how  we 
manage  risk.  Because  the  idea  several  years  ago  looking 
into  advanced  systems,  or  advanced  concepts.  We  thought 
that  airplanes  like  the  ATF  should  have  a  2-D  vectoring 
nozzle.  Having  decided  that,  then  we  had  to  look  at  what 
was  the  real  capability  to  produce  the  flying  article 
within  weight,  specs,  and  if  it  would  be  reliable.  The 
chief  engineer  would  incorporate  that  in  the  system.  A  lot 
of  features:  improved  take-off  performance — up  and  away 
maneuverability.  In  some  cases  considerations  are  embedded 
in  that.  Generally,  a  lot  of  added  capability.  There's 
clearly  some  risk  embedded  in  that  and  part  of  that  risk 
had  to  be  with  cost  and  weight  issues.  The  management  of 
that  risk  took  the  form  of  saying  ground  test  alone  is  not 
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adequate.  And  subsequently,  that  has  proven  out.  In  fact, 
Pratt  and  Whitney  did  boiler  plate — almost  feasibility 
study — and  test  cell,  put  in  a  new  engine  and  now  you  can 
see  the  thing  move  and  all  of  that.  But  as  they  tried  to 
transition  that  into  a  flight  weight  article  they 
encountered  a  lot  of  difficulties  in  manufacturing  and 
performance.  Now,  we  eventually  got  there;  the  thing  is 
flying.  But  it  is  still  not  [without  problems]  but  it  will 
establish  a  database  for  production  people  on  the  ATF 
[Advanced  Tactical  Fighter  Aircraft]  to  use.  Even  the  ones 
that  are  flying  right  now  are  not  suitable  for  putting  on  a 
production  airplane.  We're  going  to  have  to  go  one  stage 
more.  So  our  management  of  the  risk  took  the  form  of 
saying,  first  in  the  paper  studies:  this  will  help  to 
expand  the  flight  of  the  envelope  of  the  airplane.  The 
ground  test  itself  was  recognized  through  .recognize  through 
professional  judgement.  The  ground  test  itself  is  not 
adequate.  When  we  built  some  flight  articles,  that  those 
were  still  not  suitable  because  of  the  problems  we 
encounter.  They  were  still  not  suitable  as  a  production 
article.  So  there  will  be  one  more  stage  to  go  through 
this  for  the  ATF  program.  All  they  do  is  the  production  at 
the  very  end  of  that.  There  is  still  some  risk  to  that 
program  office,  that  they  may  not  meet  all  their  goals  in 
terms  of  cost,  or  weight  or  features  in  the  long  run.  But 
our  risk  management  took  the  form  of  assessing  at  each 
stage  what  we  thought  we'd  do,  what  was  positive, 
efficient,  what  was  good  to  know,  what  critical  issues  did 
we  have  to  address  at  the  next  stage.  That's  largely 
embedded  for  2-D  nozzles  or  anything.  It's  largely  embedded 
in  the  programmatic  of  the  cycle  of  POM  submittals  and 
annual  budget  reviews  of  program  advocacies.  Now  generally 
its  not  called  risk.  Risk  is  a  term  that  is  not  used  very 
much.  It  is  usually  described  more  in  traditional  things 
like:  does  this  thing  weight  too  much,  or  does  it  function 

as  well  as  it  should,  or  took  too  long  to  manufacture. 

That  is  kind  of  our  way  of  saying,  well  at  the  stage  of 
that  we  have  risk  embedded  there  because  we  didn't  [meet 
our  goals],  but  I'm  quite  sure  we  can  meet  our  goals.  The 
lab  rarely  describes  things  in  terms  of  risk  as  the 
management  schools  teach  this  king  of  favorite  subject  of 
mine.  But  it's  not  something  that  is  a  conscious  part  of 
the  way  we  do  business. 

Interviewer.  Does  you  organization  have  specific  written 
strategy  or  policy  for  assessing  and  managing  technical 
risk? 

Dr.  Brown:  First  of  all,  the  mission  statements  recognize 
the  need  to  go  through  the  sequence  that  I  just  described. 
In  order  to  eliminate  risk.  They  don't  necessarily  look  at 
a  program.  They  very  seldom  look  at  risk  typically.  I'll 
show  you  this  chart,  and  say  for  instance,  this  is  a  very 
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systematic  way  we  do  studies.  And  we  start  with  a  kind  of 
a  classical  process  for  each  program.  We  come  out  at  the 
bottom  at  the  end  having  come  out  of  a  technology 
assessment.  And  talk  about  technology  goals  and  notice 
that  risk  is  in  there.  If  you  look  at  our  formal  mission 
statements  and  the  way  we  tell  people  we  do  business  we 
hope  to  achieve  what's  in  the  risk.  Survivability 
constitutes  the  one-on-one  engagement  process.  If  I'm 
looking  at  a  penetration  system,  first  I  get  the  concept  of 
it,  then  I  look  at  the  concept  of  one-on-one  situation. 

And  get  the  piece  of  this.  Then  I  go  into  a  more 
aggregated  effectiveness  analysis  in  which  I  may  look  at 
the  total  mission  problem.  And  perhaps  the  number  of 
airplanes  dealing  with  the  number  of  threat  sites,  or  the 
number  of  supporting  systems.  Then  begin  to  understand 
what  the  total  mission  problem.  .And  perhaps  the  number  of 
airplanes  dealing  with  the  number  of  threat  sites,  or  the 
number  of  supporting  systems.  Then  begin  to  understand 
what  the  total  effectiveness  is  in  an  aggregated  sense. 
Survivability  may  mean,  as  it  flies  along  at  a  particular 
speed  at  an  altitude  at  a  particular  flight  path,  maybe 
stable,  maybe  maneuvering  and  has  a  certain  signature,  runs 
across  a  certain  encounter.  What  is  the  probability 
survival  there?  One  airplane  against  one  threat  site. 

Then  I  may  have  the  total  mission  of  reaching  the  target 
and  dropping  the  bomb  or  whatever.  Maybe  it  is  composed  of 
a  number  of  sequential  elements  that  I  have  at  the  end. 

And  the  next  step,  I'll  have  to  kind  of  total  all  of  those 
up.  And  try  to  decide  if  the  system  in  aggregate  was 
reasonably  effective. 

We  has  a  request  some  years  ago  by  Military  Airlift 
Command  to  look  at  what  might  be  done  for  C-130 
replacement.  Our  designers  started  sketching  out  what  they 
know  about  the  mission.  We  begin  to  look  at  how  the 
airplane  would  be  used  to  begin  to  refine  the  designs.  We 
looked  at  some  one-on-ones:  engagements  with  triple  A 
sites.  Also,  we  looked  at  how  we  might  vary  the 
performance  of  the  airplane.  Did  it  need  to  go  faster? 

Have  a  lower  signature?  and  begin  to  establish  those 
tradeoffs.  And  we  begin  to  compare  this  airplane  with 
perhaps  a  more  traditional  variant  of  the  transport.  All 
through  this  there  are  clearly  technologies  that  are 
required  to  make  the  airplane  feasible.  Once  we  satisfy 
ourselves  that  this  airplane  is  a  good  candidate  versus 
that.  Suppose  we  decided  that  this  is  really  better  than 
that  one  way  or  the  other.  We'd  then  say  the  reason  that 
this  was  successful  is  that  it  had  the  advance  materials, 
and  this  advance  gear  box,  and  this  advanced  crew  station. 
In  order  to  survive  an  engagement  like  this,  crew  is  going 
to  have  to  have  the  following  information  and  be  able  to 
act  on  it  expeditiously.  And  that  may  require  a  few 
displays  of  evaluating  those  in  a  simulator  or  ultimately 
perhaps  a  flight  article.  If  you  can  go  back  and  look  at 
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technology  assessment,  if  you  went  to  a  functional  division 
for  example,  if  you  went  to  a  mechanics  division  and  talk 
to  them  about  technology  assessment  they  have  figures  of 
merit  that  are  uniquely  associated  with  what  they  do  with 
their  science.  You  might  need  a  life  to  drag  [ratio  of] 
and  airfoil.  And  if  tney  can  improve  a  lift  to  drag  of  the 
airfoil,  they've  had  a  success.  We  try  to  look  at 
technology  in  terms  of  the  whole  system  and  it  may  be  that 
the  new  airfoil  does  make  some  contribution.  It  may  turn 
out,  in  fact,  that  the  real  key  to  success  is  a  redesigned 
inlet--that  is  the  most  important  thing  that  can  be  done. 
The  total  system  picture  we  would  come  back  out  with  the 
recommendation  with  the  program  and  inlets.  Then  the  guys 
down  in  XR  [ASD  Office  of  Plans]  and  other  places  may  take 
a  look  at  the  whole  scenario  of  the  war,  and  say,  what  we 
really  need  is  airplanes  that  will  fly  faster  or  something. 
They  don't  really  care  if  this  kind  of  thing  needs  a  better 
inlet  or  something.  So,  that  is  kind  of  how  we  put 
technology  into  perspective. 


207 


Appendix  P:  Interview  with  Mr.  Jack.  Cannon.  Technical 
Director  of  Development  Planning,  ASP  Office 
of  Plans.  ASD/XR.  25  July  1989 


Interviewer:  Sir,  what  are  your  technical  qualifications? 

Specifically,  what  technical  degrees  do  you  hold? 

Mr.  Cannon:  I  simply  hold  a  bachelor  in  science  with  a 
major  in  physics  with  considerable  graduate  work  above  the 
level . 

Interviewer:  What  is  your  technical  experience?  What 

technical  jobs  have  you  held? 

Mr.  Cannon:  I  worked  as  a  physicist  in  the  materials 
laboratory  for  about  five  years  and  the  rest  of  my  career 
has  been  at  the  staff  levels:  a  director  of  research  of 
Wright  Air  Development  Center,  and  later  at  ASD.  I've  been 
in  systems  planning  since  about  1963. 

Interviewer :  What  is  your  present  job? 

Mr.  Cannon:  My  position  here  is  Technical  D' rector  of 
Development  Planning  [A3D/XR] . 

Interviewer :  what  is  the  mission  of  the  organization  of 

which  you  are  a  part? 

Mr.  Cannon:  Tho  mission  of  the  organization  basically  is 
to  essentially  develop  new  system  starts — and  when  1  say 
new  system  starts  it's  not  only  totally  new  systems  but 
major  modification  to  existing  systems — to  solve  major 
command  deficiencies.  Virtually  all  of  the  systems  you 
will  find  today  came  through  this  kind  of  a  process — 
through  XR  [the  Office  of  Development  Planning] .  Things 
like  the  F-15,  the  A-10,  the  C-5,  the  B-l,  the  ATF 
[Advanced  Tactical  Fighter]  the  NASP  [National  Aero-Space 
Plane] .  All  of  those  programs  started  in  XR  and  we  take 
them  up  to  the  point  of  having  done  the  concept  exploration 
studies  and  then  transition  them  to  an  acquisition  agency, 
to  a  SPO  [Systems  Program  Office]  or  to  a  deputy  for 
acguisition. 

Interviewer:  In  your  opinion,  what  are  the  most  important 

techniques  for  managing  technical  risk?  The  is  assuming 
that  [a  system]  has  been  demonstrated  feasible  in  the 
laboratory . 

Mr.  Cannon:  You're  saying  breadboard  feasibility? 
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Interviewer :  Yes.  So  from  that  point  what  is  necessary  to 

manage  the  technical  risk? 

Mr.  Cannon:  Well,  generally,  my  feeling  would  be  that  you 
would  have  to  go  into  an  advanced  development  program  which 
we  generally  speak  of  as  being  a,  so  that  it's  prototype 
hardware  to  demonstrate  the  concept.  It  could  be 
demo.  strated  on  a  ground  test  level  in  some  cases  to 
satisfy  your  concerns  about  risk  or  it  might  have  to  be 
flight  demonstrated.  But  generally,  you  would  not  develop 
flight  qualified  hardware,  you  would  build  prototype 
hardware  that  would  be  flown  in  a  prototype  system  before 
you  would  go  the  full  nine  yards  of  fully  developing  the 
system  to  prove  out  the  concept.  You  would  prototype  it 
using  as  much  existing  technology  that  you  can  to 
demonstrate  it  before  we  get  into  new  technology  at  the 
same  time. 

Interviewer:  Based  on  your  experience,  what  is  the  most 

important  technique  for  managing  technical  risk? 

Mr.  Cannon:  Well,  you  know  if  you  had  to  select  one  I 
think  that  certainly  the  prototype  testing  is  the  way  you 
reduce  the  risk. 

Interviewer :  Does  your  organization  have  a  specific 

written  strategy  or  policy  for  assessing  and  managing 
technical  risk? 

Mr.  Cannon:  Well,  what  is  used  in  ASD  is  first  of  all,  is 
whenever  a  63XX  [budget  designation]  program  is  started,  it 
goes  through  what  is  called  the  SENTAR  process.  The  deputy 
of  engineering  of  ASD  is  really  responsible  for  the  SENTAR 
process,  what  they  do  is  review  at  the  start  of  the  63XX- 
program.  I'm  talking  about  a  advanced  development 
hardware  demonstration  program.  They  will  review  that 
proposed  advanced  development  plan,  and  particularly 
concentrate  on  setting  some  criteria  that  they  felt  should 
be  met  chrough  this  demonstration.  Some  performance  that 
would  be  demonstrated  in  the  hardware  demonstration.  Then 
once  the  laboratories  and  the  engineerix  *  people  have 
signed  up  to  that,  and  say  that  this  is  /hat  we're  going  to 
do,  and  this  is  what  the  lab  plans  to  do,  and  this  is  what 
is  going  to  be  necessary  to  do  in  order  to  satisfy  the 
engineers  that  the  technology  is  mature  and  is  documented 
in  a  transition  plan.  The  old  twist  was  basically  that  at 
the  same  time  you  involve  a  customer.  The  customer  could 
be  an  existing  SPO  or  an  existing  system  organization,  if 
there  is  no  existing  system  organization  it  then  becomes  XR 
as  the  customer  for  that  technology.  You  will  have  a  three 
prong  agreement:  a  customer  agreeing  that  it  is  useful 
technology  in  terms  of  the  future,  a  laboratory  which  is 
developing  the  technology,  and  an  engineering  organization 
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which  is  geared  to  look  over  the  results  of  the  laboratory 
program  to  make  sure  that  it  has  satisfied  the 
predetermined  criteria.  And  therefore,  the  technology 
would  be  considered  mature  at  that  time.  You  will  probably 
want  to  look  into  the  SENTAR  process,  which  stands  for 
Senior  Engineering  Technology  Assessment  Review.  It  is  a 
group  of  tech  directors  out  of  the  deputy  for  engineering, 
that  form  the  SENTAR  group,  and  they  review  all  laboratory 
63XX-programs .  The  63XX-program  stands  for  advanced 
development.  The  programs  that  are  used  in  the  Air  Force, 

6 1XX  program  is  basic  research,  62XX  program  is  exploratory 
development,  and  63XX  program  is  advanced  development.  The 
6 3 XX  program  is  broken  down  into  A&B.  63XXA  is  advanced 
technology,  and  63XXB  is  systems  oriented  development.  If 
you  we're  going  to  build  a  flying  prototype,  for  example 
prototype  it  would  be  built  under  63XXB.  Electronic 
components  would  be  built  under  63A  in  that  sort  of  broad 
technology  area.  Then  the  64XX  program  is  engineering 
development.  Ideally  something  would  flow  from  basic 
research  to  exploratory  development  to  advanced  development 
to  engineering  development.  I  say  ideally  because  you'd 
probably  find  that  many  things  don't  always  go  in  that  nice 
classical  fashion. 

Interviewer ;  Anything  else  you  would  like  to  say  about 
technical  risk? 

Mr.  Cannon:  Well,  when  we  do  advanced  studies  here  in  XR, 
I'm  talking  about  things  that  are  going  to  be  operational 
in  [the  year]  2010.  Our  first  look-see  at  technical  risk — 
we  would  be  in  our  laboratories  and  ask  for  technical  risk 
analysis.  In  other  words,  did  the  lab  see  technologies 
that  would  make  this  system  possible  in  the  year  2010? 

We've  been  able  to  describe  the  system  in  terms  of  its 
performance:  for  example,  its  speed,  range,  altitude, 

maneuver  and  so  forth.  Then  we  ask  the  laboratories 
basically  to  give  us  a  technology  assessment  in  things 
like,  materials,  structures,  avionics,  propulsion  which 
make  up  the  system,  and  they  give  us  an  initial  technical 
risk  assessment.  And  then  they  begin  to  show  that  they 
have  technology  that  they  feel  is  going  to  satisfy  that 
requirement  or  they  will  show  that  no  they  don't  have  and 
here  is  what  will  have  to  be  done  in  the  years  you  got  as 
lead  time  for  the  system  to  begin  to  develop  some 
technology.  So  I  think  that  the  key  to  us  in  this  business 
is  that  we  get  that  initial  tech  assessment  done  by  the 
laboratory  folks,  who  understand  what  technologies  they 
really  have  in  hand  and  what  they  really  have  to  do  yet  in 
order  to  satisfy  a  given  system  requirement.  As  they 
imitate  things,  then  the  SENTAR  process  begins  to  be 
involved  in  terms  of  following,  tracking  what  is  happening 
on  that  advanced  development  type  effort. 
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Interviewer:  Thank  you  a  lot  for  your  time, 

it . 


I  appreciate 
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